Planet Debian

Subscribe to Planet Debian feed
Planet Debian - https://planet.debian.org/
Updated: 15 min 57 sec ago

Jonathan Dowland: Small tweaks to `git branch` behaviour

15 July, 2021 - 15:34

Despite my best efforts, I often end up with a lot of branches in my git repositories, many of which need cleaning up, but even so, may which don't. Two git configuration tweaks make the output of git branch much more useful for me.

Motivational example, default git behaviour:

🍊git branch
  2021-apr-cpu-proposed
  OPENJDK-159-openj9-FROM
  OPENJDK-312-passwd
  OPENJDK-407-dnf-modules-fonts
  create_override_files_in_redhat_189
* develop
  inline-container-yaml
  local-modules
  mdrafiur-pr185-jolokia
  openjdk-containers-1.9
  openjdk-rm-jolokia
  osbs-openjdk
  release
  signing-intent-release
  ubi-1.3-mergedown
  ubi-11-singleton-jdk
  ubi8.2
  update-FROM-lines
  update-for-cct-module-changes-maven-etc

The default sort order is alphabetical, but that's never useful for the repositories I work in. The age of the branch is generally more useful. This particular example isn't that long, but often the number of branches can fill the screen. git can be configured to use columns for branch listings, which I think generally improves readability.

🍊git config --global branch.sort authordate
🍊git config --global column.branch auto

After:

🍊git branch
  update-for-cct-module-changes-maven-etc   signing-intent-release
  openjdk-rm-jolokia                        local-modules
  ubi8.2                                    mdrafiur-pr185-jolokia
  ubi-11-singleton-jdk                      OPENJDK-312-passwd
  ubi-1.3-mergedown                         create_override_files_in_redhat_189
  OPENJDK-159-openj9-FROM                   2021-apr-cpu-proposed
  openjdk-containers-1.9                    OPENJDK-407-dnf-modules-fonts
  inline-container-yaml                     release
  update-FROM-lines                       * develop
  osbs-openjdk

Pavit Kaur: GSoC: First Phase of Coding Period

14 July, 2021 - 10:52

Hello there.

I still can’t believe that the first half of GSoC period is almost over. So it’s been about 5 weeks working on the project and that means I have a lot to share about it. So without further ado, let’s get started.

I will be listing up my work done in the respective tasks.

Task: Migrating Logins to Salsa

The objective of this task was that the users could log in to their account on debci using their Debian Salsa account (collaborative development server for Debian based on the GitLab software) and this is implemented with the help of OmniAuth, the ruby authentication framework.

At the beginning of this, I had to discuss quite a few issues with my mentors that I was bumping into, and by the end of it with multiple revisions and discussions, the following was implemented:

  • The previous users' table schema of debci comprises the username field which contained mostly the emails of the users with some exceptions and to accommodate the Salsa logins, a new uid field is added to the table to store the Salsa uid of the logged-in user with the username field storing Salsa usernames now and as the Salsa users have the liberty to change their usernames, the updation of username as well as in debci database is also taken care of.

  • For Salsa login, the ruby-omniauth-gitlab strategy has been used and for login in development mode, the developer strategy which comes with ruby-omniauth has been set up.

  • Added a Login Page giving the option to log in using Salsa and an additional option to login in Developer Mode which is accessible only in Development Setup so that other contributors don’t have to set up dummy Salsa applications for working.

  • Added specs for the new login process. This was an interesting part, as I got the chance to understand RSpec and facilities provided by OmniAuth to mock the authentication for Integration Testing.

  • One blocker that I dealt with was that the Debian release from where packages were pulled out for debci have the OmniAuth version 1.8, which was not working well with the developer strategy implementation for the application so to resolve that I did a minor change to the callback API for developer strategy until the time that release have the newer version of OmniAuth.

  • Another thing we discussed in one of the meetings that in the existing database structure, the tests do not have a real reference to the users' table and rather the username is stored directly as a string for the requestor field, so this thing was fixed as part of this task.

The migration of the existing users' data for the new logins was handled by my mentor Antonio Terceiro and with this, our first task is concluded. All these changes are now part of Debian Continuous Integration platform and you can find the blogpost for same by Antonio here.

This task also allowed me to write my first ever tutorial Tutorial: Integrating OmniAuth with Sinatra Application to help people looking to integrate their ruby application with OmniAuth.

Moving further to the next task in progress.

Task: Adding support for testing security uploads and Debian LTS

This is the next task I am working on enabling private tests in debci for adding support for testing security uploads and Debian LTS. Since it’s a bigger task, it is broken down into about 6-7 steps and till now, the following has been done:

  • The schema of jobs' (tests) table is updated to have a boolean field to store whether the job is private or not.

  • The is_private parameter is added to both API and Self-Service section so the private test can be submitted through the API as well as through GUI form on the web platform.

  • Another thing which comes up through discussion in meetings that a parameter is required to add extra-apt-sources for getting packages of security repository and this is the part in progress.

So that concludes my work till now. It has been an amazing journey with lots of learning and also the guidance from the wonderful mentors of my project and I am looking forward to more exciting parts ahead.

That’s all for now. See you next time!

Steinar H. Gunderson: Optimization silver bullets

13 July, 2021 - 21:00

If you work with optimizing code for a while, you'll notice that a fairly common pattern is for people to believe in optimization silver bullets; just one trick that they think is always the solution for whatever woes you may have. It's not that said thing is bad per se, it's just that they keep suggesting the same thing over and over even if that's not actually the issue.

To name some examples: I've seen people suggesting removing mallocs is always the case (even if malloc didn't show up on the profile), or that adding likely() and unlikely() everywhere would double the IPC of a complex system (PGO, with near-perfect condition probabilities, gave 5%), or designed a system entirely around minimizing instruction cache pressure (where the system they intended to replace didn't have issues with instruction cache). And I guess we've all seen the people insisting on optimizing their code on -O9, because higher is better, right, and who are the GCC people to compile their own code with -O2 anyway?

I've more or less learned to ignore these people, as long as they don't show up with profiles and microbenchmarks, which they never do. (This is the easiest way to see if people's suggestions are bogeymen or real; if people know what they're doing, they can point to a real profile, and they'll write a stable microbenchmark to show that they've actually fixed the issue and to guard against future regressions.) But there's one silver bullet that always rubs me the wrong way: False sharing.

False sharing is when two unrelated items happen to lie on the same cache line, and they are accessed frequently by different cores. Seemingly, false sharing is just exotic enough that people have heard of it and are proud of that, and then they start being afraid of it everywhere for no good reason. I've seen people writing large incantations to protect against false sharing, presumably blowing the data cache in the process, and then discovered that due to them misunderstanding the compiler, the entire thing had been a no-op for years. It's pretty crazy.

That's why I was very happy to finally, after 25 years of multithreaded coding, discover a real case of false sharing in PiStorm; one thread had a local variable made global for some no-longer-relevant debugging reasons, and another thread was making constant writes to a global one in a busy loop. Really a classic, bad case of false sharing. Rune wrote up a patch, and lo and behold, the benchmarks went up!

…by about one percent.

Matthew Garrett: Does free software benefit from ML models being derived works of training data?

13 July, 2021 - 08:57
Github recently announced Copilot, a machine learning system that makes suggestions for you when you're writing code. It's apparently trained on all public code hosted on Github, which means there's a lot of free software in its training set. Github assert that the output of Copilot belongs to the user, although they admit that it may occasionally produce output that is identical to content from the training set.

Unsurprisingly, this has led to a number of questions along the lines of "If Copilot embeds code that is identical to GPLed training data, is my code now GPLed?". This is extremely understandable, but the underlying issue is actually more general than that. Even code under permissive licenses like BSD requires retention of copyright notices and disclaimers, and failing to include them is just as much a copyright violation as incorporating GPLed code into a work and not abiding by the terms of the GPL is.

But free software licenses only have power to the extent that copyright permits them to. If your code isn't a derived work of GPLed material, you have no obligation to follow the terms of the GPL. Github clearly believe that Copilot's output doesn't count as a derived work as far as US copyright law goes, and as a result the licenses on the training data don't apply to the output. Some people have interpreted this as an attack on free software - Copilot may insert code that's either identical or extremely similar to GPLed code, and claim that there are no license obligations created as a result, effectively allowing the laundering of GPLed code into proprietary software.

I'm completely unqualified to hold a strong opinion on whether Github's legal position is justifiable or not, and right now I'm also not interested in thinking about it too much. What I think is more interesting is what the impact of either position has on free software. Do we benefit more from a future where the output of Copilot (or similar projects) is considered a derived work of the training data, or one where it isn't? Having been involved in a bunch of GPL enforcement activities, it's very easy to think of this as something that weakens the GPL and, as a result, weakens free software. That was my initial reaction, but that's shifted over the past few days.

Let's look at the GNU manifesto, specifically this section:

The fact that the easiest way to copy a program is from one neighbor to another, the fact that a program has both source code and object code which are distinct, and the fact that a program is used rather than read and enjoyed, combine to create a situation in which a person who enforces a copyright is harming society as a whole both materially and spiritually; in which a person should not do so regardless of whether the law enables him to.

The GPL makes use of copyright law to ensure that GPLed work can't be taken from the commons. Anyone who produces a derived work of GPLed code is obliged to provide that work under the same terms. If software weren't copyrightable, the GPL would have no power. But this is the outcome Stallman wanted! The GPL doesn't exist because copyright is good, it exists because software being copyrightable is what enables the concept of proprietary software in the first place.

The powers that the GPL uses to enforce sharing of code are used by the authors of proprietary software to reduce that sharing. They attempt to forbid us from examining their code to determine how it works - they argue that anyone who does so is tainted, unable to contribute similar code to free software projects in case they produce a derived work of the original. Broadly speaking, the further the definition of a derived work reaches, the greater the power of proprietary software authors. If Oracle's argument that APIs are copyrightable had prevailed, it would have been disastrous for free software. If the Apple look and feel suit had established that Microsoft infringed Apple's copyright, we might be living in a future where we had no free software desktop environments.

When we argue for an interpretation of copyright law that enhances the power of the GPL, we're also enhancing the power of giant corporations with a lot of lawyers on hand. So let's look at this another way. If Github's interpretation of copyright law holds, we can train a model on proprietary code and extract concepts without having to worry about being tainted. The proprietary code itself won't enter the commons, but the ideas it embodies will. No more worries about whether you're literally copying the code that implements an algorithm you want to duplicate - simply start typing and let the model remove the risk for you.

There's a reasonable counter argument about equality here. How much GPL-influenced code is going to end up in proprietary projects when compared to the reverse? It's not an easy question to answer, but we should bear in mind that the majority of public repositories on Github aren't under an open source license. Copilot is already claiming to give us access to the concepts embodied in those repositories. Do these provide more value than is given up? I honestly don't know how to measure that. But what I do know is that free software was founded in a belief that software shouldn't be constrained by copyright, and our default stance shouldn't be to argue against the idea that copyright is weaker than we imagined.

comments

Debian XMPP Team: XMPP Novelties in Debian 11 Bullseye

13 July, 2021 - 07:00

This is not only the Year of the Ox, but also the year of Debian 11, code-named bullseye. The release lies ahead, full freeze starts this week. A good opportunity to take a look at what is new in bullseye. In this post new programs and new software versions related to XMPP, also known as Jabber are presented. XMPP exists since 1999, and has a diverse and active developers community. It is a universal communication protocol, used for instant messaging, IoT, WebRTC, and social applications. You probably will encounter some oxen in this post.

  • biboumi, XMPP gateway to connect to IRC servers: 8.3 → 9.0
    The biggest change for users is SASL support: A new field in the Configure ad-hoc command lets you set a password that will be used to authenticate to the nick service, instead of using the cumbersome NickServ method.
    Many more changes are listed in the changelog.
  • Dino, modern XMPP client: 0.0.git20181129 → 0.2.0
    Dino in Debian 10 was practically a technology preview. In Debian 11 it is already a fully usable client, supporting OMEMO encryption, file upload, image preview, message correction and many more features in a clean and beautiful user interface.
  • ejabberd, the extensible realtime platform: 18.12.1 → 21.01.
    Probably the most important improvement for end-users is XEP-0215 support to facilitate modern WebRTC-style audio/video calls. ejabberd also integrates more nicely with systemd (e.g., the watchdog feature if supported, now). Apart from that, a new configuration validator was introduced, which brings a more flexible (but mostly backwards-compatible) syntax. Also, error reporting in case of misconfiguration should be way more helpful, now. As a new authentication backend, JSON Web Tokens (JWT) can be used. In addition to the XMPP and SIP support, ejabberd now includes a full-blown MQTT server. A large number of smaller features has been added, performance was improved in many ways, and several bugs were fixed. See the long list of changes.
  • Gajim, a GTK+-based Jabber client: 1.1.2 → 1.3.1
    The new Debian release brings many improvements. Gajim’s network code has been completely rewritten, which leads to faster connections, better recovery from network loss, and less network related hiccups. Customizing Gajim is now easier than ever. Thanks to the new settings backend and a completely reworked Preferences window, you can adapt Gajim to your needs in just a few seconds.
    Good for newcomers: account creation is now a lot easier with Gajim’s new assistant. The new Profile window gives you many options to tell people more about yourself. You can now easily crop your own profile picture before updating it.
    Group chats actions have been reorganized. It’s now easier to send invitations or change your nickname for example. Gajim also received support for chat markers, which enables you to see how far your contact followed the conversation. But this is by far not everything the new release brings. There are many new and helpful features, such as pasting images from your clipboard directly into the chat or playing voice messages directly from the chat window.
    Read more about the new Gajim release in Debian 11 here.
    Furthermore, three more Gajim plugins are now in Debian: gajim-lengthnotifier, gajim-openpgp for OX 🐂 (XEP-0373: OpenPGP for XMPP) and gajim-syntaxhighlight.
  • NEW Kaidan Simple and user-friendly Jabber/XMPP client 0.7.0
    Kaidan is a simple, user-friendly and modern XMPP chat client. The user interface makes use of Kirigami and QtQuick, while the back-end of Kaidan is entirely written in C++ using Qt and the Qt-based XMPP library QXmpp. Kaidan runs on mobile and desktop systems including Linux, Windows, macOS, Android, Plasma Mobile and Ubuntu Touch.
  • mcabber, small Jabber (XMPP) console client: 1.1.0 → 1.1.2
    A theme for 256 color terminals is now included, the handling of carbon message copies has been improved, and various minor issues have been fixed.
  • Poezio, Console-based XMPP client: 0.12.1 → 0.13.1
    This new release brings many improvements, such as Message Archive (XEP-0313) support, initial support for OMEMO (XEP-0384) through a plugin, HTTP File Upload support, Consitent Color Generation (XEP-0392), and plenty of internal changes and bug fixes. Not all changes in 0.13 and 0.13.1 can be listed, see the CHANGELOG for a more extensive summary.
  • Profanity, the console based XMPP client: 0.6.0 → 0.10.0
    We can not list all changes which have been done, but here are some highlights.
    Support of OMEMO Encryption (XEP-0384). Consistent Color Generation (XEP-0392), be aware of the changes in the command to standardize the names of commands. A clipboard feature has been added. Highlight unread messages with a different color in /wins. Keyboard switch to select the next window with unread messages with alt + a. Support for Last Message Correction (XEP-0308), Allow UTF-8 symbols as OMEMO/OTR/PGP indicator char. Add option to open avatars directly (XEP-0084). Add option to define a theme at startup and some changes to improve themes. Add possibility to easily open URLs. Add experimental OX 🐂 (XEP-0373, XEP-0374) support. Add OMEMO media sharing support, ...
    There is also a Profanity light package in Debian now, the best option for systems with tight limits on resources.
  • Prosody, the lightweight extensible XMPP server: 0.11.2 → 0.11.9
    Upgrading to the latest stable release of Prosody brings a whole load of improvements in the stability, usability and performance departments. It especially improves the performance of websockets, and PEP performance for users with many contacts. It includes interoperability improvements for a range of clients.
  • prosody-modules, community modules and extensions for Prosody: 0.0~hg20190203 → 0.0~hg20210130
    The ever-growing collection of goodies to plug into Prosody has a number of exciting additions, including a suite of modules to handle invite-based account registration, and others for moderating messages in group chats (e.g. for removal of spam/abuse), server-to-server federation over Tor and client authentication using certificates. Many existing community modules received updates as well.
  • Psi, Qt-based XMPP client: 1.3 → 1.5
    The new version contains important bug fixes.
  • salutatoi, multi-frontends, multi-purposes communication tool: 0.7.0a4 → 0.8.0~hg3453
    This version is now fully running on Python 3, and has full OMEMO support (one2one, groups and files). The CLI frontend (jp) has among new commands a "jp file get" one which is comparable to wget with OMEMO support. A file sharing component is included, with HTTP Upload and Jingle support. For a list of other improvements, please consult the changelog.
    Note, that the upstream project has been renamed to "Libervia".
  • NEW sms4you, Personal gateway connecting SMS to XMPP or email 0.0.7
    It runs with a GSM device over ModemManager and uses a lightweight XMPP server or a single email account to handle communication in both directions.
  • NEW xmppc, XMPP Command Line Client 0.1.0
    xmppc is a new command line tool for XMPP. It supports some basic features of XMPP (request your roster, bookmarks, OMEMO Devices and fingerprints). You can send messages with both legacy PGP (XEP-0027) and the new OX 🐂 (XEP-0373: OpenPGP for XMPP).

That's all for now. Enjoy Debian 11 bullseye and Happy Chatting!

Chris Lamb: Saint Alethia? On Bodies of Light by Sarah Moss

13 July, 2021 - 01:10

How are you meant to write about an unfinished emancipation? Bodies of Light is a 2014 book by Glasgow-born Sarah Moss on the stirrings of women's suffrage in an arty clique in nineteenth-century England. Set in the intellectually smoggy cities of Manchester and London, we follow the studious and intelligent Alethia 'Ally' Moberly, who is struggling to gain the acceptance of herself, her mother and the General Medical Council.

'Alethia' may be the Greek goddess of truth, but our Ally is really searching for wisdom. Her strengths are her patience and bookish learning, and she acquires Latin as soon as she learns male doctors will use it to keep women away from the operating theatre. In fact, Ally's acquisition of language becomes a recurring leitmotif: replaying a suggestive dream involving a love interest, for instance, Ally thinks of 'dark, tumbling dreams for which she has a perfectly adequate vocabulary'. There are very few moments of sensuality in the book, and pairing it with Ally's understated wit achieves a wonderful effect.

The amount we learn about a character is adapted for effect as well. There are few psychological insights about Ally's sister, for example, and she thus becomes a fey, mysterious and almost Pre-Raphaelite figure below the surface of a lake to match the artistic movement being portrayed. By contrast, we get almost the complete origin story of Ally's mother, Elizabeth, who also constitutes of those rare birds in literature: an entirely plausible Christian religious zealot. Nothing Ally does is ever enough for her, but unlike most modern portrayals of this dynamic, neither of them are aware of what is going, and it is conveyed in a way that is chillingly... benevolent. This was brought home in the annual 'birthday letters' that Elizabeth writes to her daughter:

Last year's letter said that Ally was nervous, emotional and easily swayed, and that she should not allow her behaviour to be guided by feeling but remember always to assert her reason. Mamma would help her with early hours, plain food and plenty of exercise. Ally looks at the letter, plump in its cream envelope. She hopes Mamma wrote it before scolding her yesterday.

§

The book makes the implicit argument that it is a far more robust argument against pervasive oppression to portray a character in, say, 'a comfortable house, a kind husband and a healthy child', yet they are nonetheless still deeply miserable, for reasons they can't quite put their finger on. And when we see Elizabeth perpetuating some generational trauma with her own children, it is telling that is pattern is not short-circuited by an improvement in their material conditions. Rather, it is arrested only by a kind of political consciousness — in Ally's case, the education in a school. In fact, if there is a real hero in Bodies of Light, it is the very concept of female education.

There's genuine shading to the book's ideological villains, despite finding their apotheosis in the jibes about 'plump Tories'. These remarks first stuck out to me as cheap thrills by the author; easy and inexpensive potshots that are unbecoming of the pages around them. But they soon prove themselves to be moments of much-needed humour. Indeed, when passages like this are read in their proper context, the proclamations made by sundry Victorian worthies start to serve as deadpan satire:

We have much evidence that the great majority of your male colleagues regard you as an aberration against nature, a disgusting, unsexed creature and a danger to the public.

Funny as these remarks might be, however, these moments have a subtler and more profound purpose as well. Historical biography always has the risk of allowing readers to believe that the 'issue' has already been solved — hence, perhaps, the enduring appeal of science fiction. But Moss providing these snippets from newspapers 150 years ago should make a clear connection to a near-identical moral panic today.

§

On the other hand, setting your morality tale in the past has the advantage that you can show that progress is possible. And it can also demonstrate how that progress might come about as well. This book makes the argument for collective action and generally repudiates individualisation through ever-fallible martyrs. Ally always needs 'allies' — not only does she rarely work alone, but she is helped in some way by almost everyone around her. This even includes her rather problematic mother, forestalling any simplistic proportioning of blame. (It might be ironic that Bodies of Light came out in 2014, the very same year that Sophia Amoruso popularised the term 'girl boss'.) Early on, Ally's schoolteacher is coded as the primary positive influence on her, but Ally's aunt later inherits this decisive role, continuing Ally's education on cultural issues and what appears to be the Victorian version of 'self-care'. Both the aunt and the schoolteacher are, of course, surrogate mother figures.

After Ally arrives in the cut-throat capital, you often get the impression you are being shown discussions where each of the characters embodies a different school of thought within first-wave feminism. This can often be a fairly tedious device in fiction, the sort of thing you would find in a Sally Rooney novel, Pilgrim's Progress or some other ponderously polemical tract. Yet when Ally appears to 'win' an argument, it is only in the sense that the narrator continues to follow her, implicitly and lightly endorsing her point. Perhaps if I knew my history better, I might be able to associate names with the book's positions, but perhaps it is better (at least for the fiction-reading experience...) that I don't, as the baggage of real-world personalities can often get in the way. I'm reminded here of Regina King's One Night in Miami... (2020), where caricatures of Malcolm X, Muhammad Ali, Jim Brown and Sam Cooke awkwardly replay various arguments within an analogous emancipatory struggle.

Yet none of the above will be the first thing a reader will notice. Each chapter begins with a description of an imaginary painting, providing a title and a date alongside a brief critical exegesis. The artworks serve a different purpose in each chapter: a puzzle to be unlocked, a fear to be confirmed, an unsolved enigma. The inclusion of (artificial) provenances is interesting as well, not simply because they add colour and detail to the chapter to come, but because their very inclusion feels reflective of how we see art today.

Orphelia (1852) by Sir John Everett Millais.

To continue the question this piece began, how should an author conclude a story about an as-yet-unfinished struggle for emancipation? How can they? Moss' approach dares you to believe the ending is saccharine or formulaic, but what else was she meant to turn in — yet another tale of struggle and suffering? After all, Thomas Hardy has already written Tess of the d'Urbervilles. All the same, it still feels slightly unsatisfying to end merely with Ally's muted, uncelebrated success.

Nevertheless, I suspect many readers will dislike the introduction of a husband in the final pages, taking it as a betrayal of the preceding chapters. Yet Moss denies us from seeing the resolution as a Disney-style happy ending. True, Ally's husband turns out to be a rather dashing lighthouse builder, but isn't it Ally herself who is lighting the way in their relationship, warning other women away from running aground on the rocks of mental illness? And Tom feels more of a reflection of Ally's newly acquired self-acceptance instead of that missing piece she needed all along. We learn at one point that Tom's 'importance to her is frightening' — this is hardly something a Disney princess would say.

In fact, it is easy to argue that a heroic ending for Ally might have been an even more egregious betrayal. The evil of saints is that you can never live up to them, for the concept of a 'saint' embodies an unreachable ideal that no human can begin to copy. By being taken as unimpeachable and uncorrectable as well, saints preclude novel political action, and are therefore undoubtedly agents of reaction. Appreciating historical figures as the (flawed) people that they really were is the first step if you wish to continue — or adapt — their political ideas.

§


I had acquired Bodies of Light after enjoying Moss' Summerwater (2020), which had the dubious honour of being touted as the 'first lockdown novel', despite it being finished before Covid-19. There are countless ways one might contrast the two, so I will limit myself to the sole observation that the strengths of one are perhaps the weaknesses of the other. It's not that Bodies of Light ends with a whimper, of course, as it quietly succeeds in concert with Ally. But by contrast, the tighter arc of Summerwater (which is set during a single day, switches protagonist between chapters, features a closed-off community, etc.) can reach a higher high with its handful of narrative artifices. Summerwater is perhaps like Phil Collins' solo career: 'more satisfying, in a narrower way.'

Daniel Silverstone: Subplot - First public alpha release

13 July, 2021 - 00:06

This weekend we (Lars and I) finished our first public alpha release of Subplot. Subplot is a tool for helping you to document your acceptance criteria for a project in such a way that you can also produce a programmatic test suite for the verification criteria. We centre this around the concept of writing a Markdown document about your project, with the option to write Gherkin-like given/when/then scenarios inside which detail the automated verification of the acceptance criteria.

This may sound very similar to Yarn, a similar concept which Lars, Richard, and I came up with in 2013. Critically back then we were very 'software engineer' focussed and so Yarn was a testing tool which happened to also produce reasonable documentation outputs if you squinted sideways and tried not to think too critically about them. Subplot on the other hand considers the documentation output to be just as important, if not more important, than the test suite output. Yarn was a tool which ran tests embedded in Markdown files, where Subplot is a documentation tool capable of extracting tests from an acceptance document for use in testing your project.

The release we made is the first time we're actively asking other people to try Subplot and see whether the concept is useful to them. Obviously we expect there to be plenty of sharp corners and there's a good amount of functionality yet to implement to make Subplot as useful as we want it to be, but if you find yourself looking at a project and thinking "How do I make sure this is acceptable to the stakeholders without first teaching them how to read my unit tests?" then Subplot may be the tool for you.

While Subplot can be used to produce test suites with functions written in Bash, Python, or Rust, the only language we're supporting as first-class in this release is Python. However I am personally most interested in the Rust opportunity as I see a lot of Rust programs very badly tested from the perspective of 'acceptance' as there is a tendency in Rust projects to focus on unit-type tests. If you are writing something in Rust and want to look at producing some high level acceptance criteria and yet still test in Rust, then please take a look at Subplot, particularly how we test subplotlib itself.

Issues, feature requests, and perhaps most relevantly, code patches, gratefully received. A desire to be actively involved in shaping the second goal of Subplot even more so.

Laura Arjona Reina: Android backups with rsync

11 July, 2021 - 02:19

A quick note to self to remind how I do backups of my Android device with rsync (and adb).

I have followed this guide: How to use rsync over USB on Android with adb

My personal notes:

  • I have Lineage so I have rsync in my Android device already installed
  • I run Debian stable (buster, for now) on my laptop, with adb installed
  • My /sdcard/rsyncd.conf file:

address = 127.0.0.1
port = 1873
uid = 0
gid = 0
[root]
path = /
use chroot = false
read only = false'

  • The command:

adb shell /data/local/tmp/rsync --daemon --no-detach --config=/sdcard/rsyncd.conf --log-file=/proc/self/fd/2

didn't work, produced this message: "@ERROR: protocol startup error" so I ended up doing:

adb shell
rsync --daemon --no-detach --config=/sdcard/rsyncd.conf --log-file=/sdcard/rsync.log

and opened another tab to perform the rsync commands from my laptop:

rsync -av --progress --stats rsync://localhost:6010/root/storage .
rsync -av --progress --stats rsync://localhost:6010/root/data .

Then I saw that rsync was copying the symlinks instead of their contents: /storage/self/primary was a broken link to /mnt/user/0/primary

So I ran again the commands with -LK:

rsync -av --progress --stats -LK rsync://localhost:6010/root/storage .
rsync -av --progress --stats -LK rsync://localhost:6010/root/data .

and now I have a copy of all the files I'm interested. In addition to this, I run an adb backup of the system:

adb backup -f ./adb_backup_apk_shared_all_system.ad -apk -shared -all -system

and I think that's all that I need for the case I want to remove stuff from my phone or some disaster happens.

Joey Hess: a bitter pill for Microsoft Copilot

10 July, 2021 - 21:19

These blackberries are so sweet and just out there in the commons, free for the taking. While picking a gallon this morning, I was thinking about how neat it is that Haskell is not one programming language, but a vast number of related languages. A lot of smart people have, just for fun, thought of ways to write Haskell programs that do different things depending on the extensions that are enabled. (See: Wait, what language is this?)

I've long wished for an AI to put me out of work programming. Or better, that I could collaborate with. Haskell's type checker is the closest I've seen to that but it doesn't understand what I want. I always imagined I'd support citizenship a full, general AI capable of that. I did not imagine that the first real attempt would be the product of a rent optimisation corporate AI, that throws all our hard work in a hopper, and deploys enough lawyers to muddy the question of whether that violates our copyrights.

Perhaps it's time to think about non-copyright mitigations. Here is an easy way, for Haskell developers. Pick an extension and add code that loops when it's not enabled. Or when it is enabled. Or when the wrong combination of extensions are enabled.

{-# LANGUAGE NumDecimals #-}

main :: IO ()
main = if show(1e1) /= "10" then main else do

I will deploy this mitigation in my code where I consider it appropriate. I will not be making my code do anything worse than looping, but of course this method could be used to make Microsoft Copilot generate code that is as problimatic as necessary.

Dirk Eddelbuettel: drat 0.2.1: Small Tweak

10 July, 2021 - 18:02

A new minor release of drat arrived on CRAN overnight. This is a minor update relative to the 0.2.0 release in April. This release will now create an empty file index.html in the top-level (when initRepo() is called), and check for presence of such a file when adding files to a repo (via insertPackage()). This helps to avoid getting ‘404’ results when (perfectly valid) drat repos are checking by accessing the top-level URL, as for example CRAN does when testing if an Additional_repositoiries is reachable. The ‘step-by-step’ vignette had already suggested creating one by hand, this is now done programmatically (and one is present in the repo suggsted to fork from too).

drat stands for drat R Archive Template, and helps with easy-to-create and easy-to-use repositories for R packages. Since its inception in early 2015 it has found reasonably widespread adoption among R users because repositories with marked releases is the better way to distribute code. See below for a few custom reference examples.

Because for once it really is as your mother told you: Friends don’t let friends install random git commit snapshots. Properly rolled-up releases it is. Just how CRAN shows us: a model that has demonstrated for two-plus decades how to do this. And you can too: drat is easy to use, documented by six vignettes and just works.

The NEWS file summarises the release as follows:

Changes in drat version 0.2.1 (2021-07-09)
  • Two internal functions now have a note in their documentation stating them as not exported (Dirk in response to #123)

  • Repositories created by initRepo now have an placeholder index.html to not trigger a curl check at CRAN (Dirk)

  • Adding to a repository now checks for a top-level index.html and displays a message if missing (Dirk)

  • The DratStepByStep.Rmd vignette mentions the added index.html file

Courtesy of my CRANberries, there is a comparison to the previous release. More detailed information is on the drat page.

If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Junichi Uekawa: Fixed multi-track audio recording web page that was broken for a while.

10 July, 2021 - 16:04
Fixed multi-track audio recording web page that was broken for a while. here. But I think I wasn't satisfied with the synchronization.

Sean Whitton: Live replacement of provider cloud images with upstream Debian

10 July, 2021 - 11:20

Tonight I’m provisioning a new virtual machine at Hetzner and I wanted to share how Consfigurator is helping with that. Hetzner have a Debian “buster” image you can start with, as you’d expect, but it comes with things like cloud-init, preconfiguration to use Hetzner’s apt mirror which doesn’t serve source packages(!), and perhaps other things I haven’t discovered. It’s a fine place to begin, but I want all the configuration for this server to be explicit in my Consfigurator consfig, so it is good to start with pristine upstream Debian. I could boot one of Hetzner’s installation ISOs but that’s slow and manual. Consfigurator can replace the OS in the VM’s root filesystem and reboot for me, and we’re ready to go.

Here’s the configuration:

(defhost foo.silentflame.com (:deploy ((:ssh :user "root") :sbcl))
  (os:debian-stable "buster" :amd64)

  ;; Hetzner's Debian 10 image comes with a three-partition layout and boots
  ;; with traditional BIOS.
  (disk:has-volumes
   (physical-disk
    :device-file "/dev/sda" :boots-with '(grub:grub :target "i386-pc")))

  (on-change (installer:cleanly-installed-once
              nil
              ;; This is a specification of the OS Hetzner's image has, so
              ;; Consfigurator knows how to install SBCL and debootstrap(8).
              ;; In this case it's the same Debian release as the replacement.
              '(os:debian-stable "buster" :amd64))

    ;; Clear out the old OS's EFI system partition contents, in case we can
    ;; switch to booting with EFI at some point (if we wanted we could specify
    ;; an additional x86_64-efi target above, and grub-install would get run
    ;; to repopulate /boot/efi, but I don't think Hetzner can boot from it yet).
    (file:directory-does-not-exist "/boot/efi/EFI")

    (apt:installed "linux-image-amd64")
    (installer:bootloaders-installed)

    (fstab:entries-for-volumes
     (disk:volumes
       (mounted-ext4-filesystem :mount-point "/")
       (partition
        (mounted-fat32-filesystem
         :mount-options '("umask=0077") :mount-point "/boot/efi"))))
    (file:lacks-lines "/etc/fstab" "# UNCONFIGURED FSTAB FOR BASE SYSTEM")

    (file:is-copy-of "/etc/resolv.conf" "/old-os/etc/resolv.conf")
    (mount:unmounted-below-and-removed "/old-os"))

  (apt:mirror "http://ftp.de.debian.org/debian")
  (apt:no-pdiffs)
  (apt:standard-sources.list)
  (sshd:installed)
  (as "root" (ssh:authorized-keys +spwsshkey+))
  (sshd:no-passwords)
  (timezone:configured "Etc/UTC")
  (swap:has-swap-file "2G")

  (network:clean-/etc/network/interfaces)
  (network:static "enp1s0" "xxx.xxx.xxx.xxx" "xxx.xxx.1.1" "255.255.255.255"))

and to use it you evaluate this at the REPL:

CONSFIG> (deploy ((:ssh :user "root" :hop "xxx.xxx.xxx.xxx") :sbcl) foo.silentflame.com)

Here the :HOP parameter specifies the IP address of the new machine, as DNS hasn’t been updated yet. Consfigurator installs SBCL and debootstrap(8), prepares a minimal system, replaces the contents of /, gets to work applying the other properties, and then reboots. This gets us a properly populated fstab:

UUID=...            /           ext4    relatime    0   1
PARTUUID=...        /boot/efi   vfat    umask=0077  0   2
/var/lib/swapfile   swap        swap    defaults    0   0

(slightly doctored for more readable alignment)

There’s ordering logic so that the swapfile will end up after whatever filesystem contains it; a UUID is used for ext4 filesystems, but for fat32 filesystems, to be safe, a PARTUUID is used.

The application of (INSTALLER:BOOTLOADERS-INSTALLED) handles calling both update-grub(8) and grub-install(8), relying on the metadata specified about /dev/sda. Next time we execute Consfigurator against the machine, it’ll ignore all the property applications attached to the application of (INSTALLER:CLEANLY-INSTALLED-ONCE) with ON-CHANGE, and just apply everything following that block.

There are a few things I don’t have good solutions for. When you boot Hetzner’s image the primary network interface is eth0, but then for a freshly debootstrapped Debian you get enp1s0, and I haven’t got a good way of knowing what it’ll be (if you know it’ll have the same name, you can use (NETWORK:PRESERVE-STATIC-ONCE) to create a file in /etc/network/interfaces.d based on the current default route and corresponding interface).

Another tricky thing is SSH host keys. It’s easy to use Consfigurator to add host keys to your laptop’s ~/.ssh/known_hosts, but in this case the host key changes back and forth from whatever the Hetzner image has and the newly generated key you get afterwards. One option might be to copy the old host keys out of /old-os before it gets deleted, like how /etc/resolv.conf is copied.

This work is based on Propellor’s equivalent functionality. I think my approach to handling /etc/fstab and bootloader installation is an improvement on what Joey does.

Pages

Creative Commons License ลิขสิทธิ์ของบทความเป็นของเจ้าของบทความแต่ละชิ้น
ผลงานนี้ ใช้สัญญาอนุญาตของครีเอทีฟคอมมอนส์แบบ แสดงที่มา-อนุญาตแบบเดียวกัน 3.0 ที่ยังไม่ได้ปรับแก้