Planet Debian

Subscribe to Planet Debian feed
Planet Debian - https://planet.debian.org/
Updated: 1 hour 39 min ago

Dirk Eddelbuettel: x13binary 1.1.57-1 on CRAN: New Upstream, New M1 Binary

21 hours 30 min ago

Christoph and I are please to share that a new release 1.1.57-1 of x13binary, of the X-13ARIMA-SEATS program by the US Census Bureau (with updated upstream release 1.1.57) is now on CRAN.

The x13binary package takes the pain out of installing X-13ARIMA-SEATS by making it a fully resolved CRAN dependency. For example, when installing the excellent seasonal package by Christoph, then X-13ARIMA-SEATS will get pulled in via the x13binary package and things just work. Just depend on x13binary and on all major OSs supported by R you should have an X-13ARIMA-SEATS binary installed which will be called seamlessly by the higher-level packages such as seasonal or gunsales. With this the full power of the what is likely the world’s most sophisticated deseasonalization and forecasting package is now at your fingertips and the R prompt, just like any other of the 17960+ CRAN packages. You can read more about this (and the seasonal package) in the Journal of Statistical Software paper by Christoph and myself.

This release brings a new upstream release as well as binaries. We continue to support two Linux flavours (theh standard x86_64 as well as armv7l), windows and for a first time two macOS flavour. In addition to the existing Intel binary we now have a native built using the arm64 “M1” chip (with thanks to Kirill for the assist).

We still lack a genuine binary for Solaris so if any of the esteemed readers of this post happens to have access to R on Solaris along with a basic Fortran compiler, we would love to hear from you. Building X-13ARIMA-SEATS from source on Solaris should be straightforward as it is on the other OSs. Or is someone with a bit of time wants to help following Gabor’s tutorial we would greatly appreciate it.

Courtesy of my CRANberries, there is also a diffstat report for this release showing changes to the previous release.

If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Petter Reinholdtsen: Mechanic's words in five languages, English, Norwegian and Northern Sámi editions

4 August, 2021 - 20:30

Almost thirty years ago, some forward looking people interested in metal work and Northern Sámi, decided to create a list of words used in Northern Sámi metal work. After almost ten years this resulted in a dictionary database, published as the book "Mekanihkkársánit : Mekanikerord = Mekaanisen alan sanasto = Mechanic's words" in 1999. The story of this work is available from the pen of Svein Lund, one of the leading actors behind this effort. They even got the dictionary approved by the Sámi Parliament of Norway as the recommended metal work words to use.

Fast forward twenty years, I came across this work when I recently became interested in metal work, and started watching educational and funny videos on the topic, like the ones from mrpete222 and This Old Tony. But they all talk English, but I wanted to know what the tools and techniques they used were called in Norwegian. Trying to track down a good dictionary from English to Norwegian, after much searching, I came across the database of words created almost thirty years ago, with translations into English, Norwegian, Northern Sámi, Swedish and Finnish. This gave me a lot of the Norwegian phrases I had been looking for. To make it easier for the next person trying to track down a good Norwegian dictionary for the metal worker, and because I knew the person behind the database from my Skolelinux / Debian Edu days, I decided to ask if the database could be released to the public without any usage limitations, in other words as a Creative Commons licensed data set. And happily, after consulting with the Sámi Parliament of Norway, the database is now available with the Creative Commons Attribution 4.0 International license from my gitlab repository.

The dictionary entries look slightly different, depending on the language in focus. This is the same entry in the different editions.

English

lathe

dreiebenk (nb) várve, várvenbeaŋka, jorahanbeaŋka, vátnanbeaŋka (se) svarv (sv) sorvi (fi)

Norwegian

dreiebenk

lathe (en) várve, várvenbeaŋka, jorahanbeaŋka, vátnanbeaŋka (se) svarv (sv) sorvi (fi)

(nb): sponskjærande bearbeidingsmaskin der ein med skjæreverktøy lausgjør spon frå eit roterande arbetsstykke

Northern Sámi

várve, várvenbeaŋka, jorahanbeaŋka, vátnanbeaŋka

dreiebenk (nb) lathe (en) svarv (sv) sorvi (fi)

(se): mašiidna mainna čuohppá vuolahasaid jorri bargoávdnasis

(nb): sponskjærande bearbeidingsmaskin der ein med skjæreverktøy lausgjør spon frå eit roterande arbetsstykke

The database included term description in both Norwegian and Northern Sámi, but not English. Because of this, the Northern Sámi edition include both descriptions, the Norwegian edition include the Norwegian description and the English edition lack a descripiton.

Once the database was available without any usage restrictions, and armed with my experience in publishing books, I decided to publish a Norwegian/English dictionary as a book using the database, to make the data set available also on paper and as an ebook. Further into the project, it occurred to me that I could just as easily make an English dictionary, and talking to Svein and concluding that it was within reach, I decided to make a Northern Sámi dictionary too.

Thus I suddenly find myself publishing a Northern Sámi dictionary, even though I do not understand the language myself. I hope it will be well received, and can help revive the impressive work done almost thirty years ago to document the vocabulary of metal workers. If I get some help, I might even extend it with some of the words I find missing, like collet, rotary broach, carbide, knurler, arbor press and others. But the first edition build from a lightly edited version of the original database, with no new entries added. If you would like to check it out, visit my list of published books and consider buying a paper or ebook copy from lulu.com. The paper edition is only available in hardcover to increase its durability in the workshop.

I am very happy to report that in the process, and thanks to help from both Svein Lund and Børre Gaup who understand the language, the docbook tools I use to create books, dblatex and docbook-xsl, now include support for Northern Sámi. Before I started, these lacked the needed locale settings for this language, but now the patches are included upstream.

Junichi Uekawa: Wrote a tool to parse /sys/block/*/stat.

4 August, 2021 - 10:40
Wrote a tool to parse /sys/block/*/stat. It's probably impossible for a human brain to appreciate the numbers so I made a web page that you can paste the contents and parse it from JS to emit some processed numbers. Probably iostat is the tool you want, but hey, sometimes you need this kind of stuff.

Dirk Eddelbuettel: RcppFarmHash 0.0.2: Maintenance

3 August, 2021 - 06:55

A minor maintenance release of the new package RcppFarmHash, first released in version 0.0.1 a week ago, is now on CRAN in an version 0.0.2.

RcppFarmHash wraps the Google FarmHash family of hash functions (written by Geoff Pike and contributors) that are used for example by Google BigQuery for the FARM_FINGERPRINT digest.

This releases adds a #define which was needed on everybody’s favourite CRAN platform to not attempt to include a missing header endian.h. With this added #define all is well as we can already tell from looking at the CRAN status where the three machines maintained by you-may-know-who have already built the package. The others will follow over the next few days.

I also tweeted about the upload with a screenshot demonstrating an eight minute passage from upload to acceptance with the added #ThankYouCRAN tag to say thanks for very smooth and fully automated processing at their end.

The very brief NEWS entry follows:

Changes in version 0.0.2 (2021-08-02)
  • On SunOS, set endianness to not error on #include endian.h

  • Add badges and installation notes to README as package is on CRAN

If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Colin Watson: Launchpad now runs on Python 3!

2 August, 2021 - 17:34

After a very long porting journey, Launchpad is finally running on Python 3 across all of our systems.

I wanted to take a bit of time to reflect on why my emotional responses to this port differ so much from those of some others who’ve done large ports, such as the Mercurial maintainers. It’s hard to deny that we’ve had to burn a lot of time on this, which I’m sure has had an opportunity cost, and from one point of view it’s essentially running to stand still: there is no single compelling feature that we get solely by porting to Python 3, although it’s clearly a prerequisite for tidying up old compatibility code and being able to use modern language facilities in the future. And yet, on the whole, I found this a rewarding project and enjoyed doing it.

Some of this may be because by inclination I’m a maintenance programmer and actually enjoy this sort of thing. My default view tends to be that software version upgrades may be a pain but it’s much better to get that pain over with as soon as you can rather than trying to hold back the tide; you can certainly get involved and try to shape where things end up, but rightly or wrongly I can’t think of many cases when a righteously indignant user base managed to arrange for the old version to be maintained in perpetuity so that they never had to deal with the new thing (OK, maybe Perl 5 counts here).

I think a more compelling difference between Launchpad and Mercurial, though, may be that very few other people really had a vested interest in what Python version Launchpad happened to be running, because it’s all server-side code (aside from some client libraries such as launchpadlib, which were ported years ago). As such, we weren’t trying to do this with the internet having Strong Opinions at us. We were doing this because it was obviously the only long-term-maintainable path forward, and in more recent times because some of our library dependencies were starting to drop support for Python 2 and so it was obviously going to become a practical problem for us sooner or later; but if we’d just stayed on Python 2 forever then fundamentally hardly anyone else would really have cared directly, only maybe about some indirect consequences of that. I don’t follow Mercurial development so I may be entirely off-base, but if other people were yelling at me about how late my project was to finish its port, that in itself would make me feel more negatively about the project even if I thought it was a good idea. Having most of the pressure come from ourselves rather than from outside meant that wasn’t an issue for us.

I’m somewhat inclined to think of the process as an extreme version of paying down technical debt. Moving from Python 2.7 to 3.5, as we just did, means skipping over multiple language versions in one go, and if similar changes had been made more gradually it would probably have felt a lot more like the typical dependency update treadmill. I appreciate why not everyone might want to think of it this way: maybe this is just my own rationalization.

Reflections on porting to Python 3

I’m not going to defend the Python 3 migration process; it was pretty rough in a lot of ways. Nor am I going to spend much effort relitigating it here, as it’s already been done to death elsewhere, and as I understand it the core Python developers have got the message loud and clear by now. At a bare minimum, a lot of valuable time was lost early in Python 3’s lifetime hanging on to flag-day-type porting strategies that were impractical for large projects, when it should have been providing for “bilingual” strategies (code that runs in both Python 2 and 3 for a transitional period) which is where most libraries and most large migrations ended up in practice. For instance, the early advice to library maintainers to maintain two parallel versions or perhaps translate dynamically with 2to3 was entirely impractical in most non-trivial cases and wasn’t what most people ended up doing, and yet the idea that 2to3 is all you need still floats around Stack Overflow and the like as a result. (These days, I would probably point people towards something more like Eevee’s porting FAQ as somewhere to start.)

There are various fairly straightforward things that people often suggest could have been done to smooth the path, and I largely agree: not removing the u'' string prefix only to put it back in 3.3, fewer gratuitous compatibility breaks in the name of tidiness, and so on. But if I had a time machine, the number one thing I would ask to have been done differently would be introducing type annotations in Python 2 before Python 3 branched off. It’s true that it’s technically possible to do type annotations in Python 2, but the fact that it’s a different syntax that would have to be fixed later is offputting, and in practice it wasn’t widely used in Python 2 code. To make a significant difference to the ease of porting, annotations would need to have been introduced early enough that lots of Python 2 library code used them so that porting code didn’t have to be quite so much of an exercise of manually figuring out the exact nature of string types from context.

Launchpad is a complex piece of software that interacts with multiple domains: for example, it deals with a database, HTTP, web page rendering, Debian-format archive publishing, and multiple revision control systems, and there’s often overlap between domains. Each of these tends to imply different kinds of string handling. Web page rendering is normally done mainly in Unicode, converting to bytes as late as possible; revision control systems normally want to spend most of their time working with bytes, although the exact details vary; HTTP is of course bytes on the wire, but Python’s WSGI interface has some string type subtleties. In practice I found myself thinking about at least four string-like “types” (that is, things that in a language with a stricter type system I might well want to define as distinct types and restrict conversion between them): bytes, text, “ordinary” native strings (str in either language, encoded to UTF-8 in Python 2), and native strings with WSGI’s encoding rules. Some of these are emergent properties of writing in the intersection of Python 2 and 3, which is effectively a specialized language of its own without coherent official documentation whose users must intuit its behaviour by comparing multiple sources of information, or by referring to unofficial porting guides: not a very satisfactory situation. Fortunately much of the complexity collapses once it becomes possible to write solely in Python 3.

Some of the difficulties we ran into are not ones that are typically thought of as Python 2-to-3 porting issues, because they were changed later in Python 3’s development process. For instance, the email module was substantially improved in around the 3.2/3.3 timeframe to handle Python 3’s bytes/text model more correctly, and since Launchpad sends quite a few different kinds of email messages and has some quite picky tests for exactly what it emits, this entailed a lot of work in our email sending code and in our test suite to account for that. (It took me a while to work out whether we should be treating raw email messages as bytes or as text; bytes turned out to work best.) 3.4 made some tweaks to the implementation of quoted-printable encoding that broke a number of our tests in ways that took some effort to fix, because the tests needed to work on both 2.7 and 3.5. The list goes on. I got quite proficient at digging through Python’s git history to figure out when and why some particular bit of behaviour had changed.

One of the thorniest problems was parsing HTTP form data. We mainly rely on zope.publisher for this, which in turn relied on cgi.FieldStorage; but cgi.FieldStorage is badly broken in some situations on Python 3. Even if that bug were fixed in a more recent version of Python, we can’t easily use anything newer than 3.5 for the first stage of our port due to the version of the base OS we’re currently running, so it wouldn’t help much. In the end I fixed some minor issues in the multipart module (and was kindly given co-maintenance of it) and converted zope.publisher to use it. Although this took a while to sort out, it seems to have gone very well.

A couple of other interesting late-arriving issues were around pickle. For most things we normally prefer safer formats such as JSON, but there are a few cases where we use pickle, particularly for our session databases. One of my colleagues pointed out that I needed to remember to tell pickle to stick to protocol 2, so that we’d be able to switch back and forward between Python 2 and 3 for a while; quite right, and we later ran into a similar problem with marshal too. A more surprising problem was that datetime.datetime objects pickled on Python 2 require special care when unpickling on Python 3; rather than the approach that ended up being implemented and documented for Python 3.6, though, I preferred a custom unpickler, both so that things would work on Python 3.5 and so that I wouldn’t have to risk affecting the decoding of other pickled strings in the session database.

General lessons

Writing this over a year after Python 2’s end-of-life date, and certainly nowhere near the leading edge of Python 3 porting work, it’s perhaps more useful to look at this in terms of the lessons it has for other large technical debt projects.

I mentioned in my previous article that I used the approach of an enormous and frequently-rebased git branch as a working area for the port, committing often and sometimes combining and extracting commits for review once they seemed to be ready. A port of this scale would have been entirely intractable without a tool of similar power to git rebase, so I’m very glad that we finished migrating to git in 2019. I relied on this right up to the end of the port, and it also allowed for quick assessments of how much more there was to land. git worktree was also helpful, in that I could easily maintain working trees built for each of Python 2 and 3 for comparison.

As is usual for most multi-developer projects, all changes to Launchpad need to go through code review, although we sometimes make exceptions for very simple and obvious changes that can be self-reviewed. Since I knew from the outset that this was going to generate a lot of changes for review, I therefore structured my work from the outset to try to make it as easy as possible for my colleagues to review it. This generally involved keeping most changes to a somewhat manageable size of 800 lines or less (although this wasn’t always possible), and arranging commits mainly according to the kind of change they made rather than their location. For example, when I needed to fix issues with / in Python 3 being true division rather than floor division, I did so in one commit across the various places where it mattered and took care not to mix it with other unrelated changes. This is good practice for nearly any kind of development, but it was especially important here since it allowed reviewers to consider a clear explanation of what I was doing in the commit message and then skim-read the rest of it much more quickly.

It was vital to keep the codebase in a working state at all times, and deploy to production reasonably often: this way if something went wrong the amount of code we had to debug to figure out what had happened was always tractable. (Although I can’t seem to find it now to link to it, I saw an account a while back of a company that had taken a flag-day approach instead with a large codebase. It seemed to work for them, but I’m certain we couldn’t have made it work for Launchpad.)

I can’t speak too highly of Launchpad’s test suite, much of which originated before my time. Without a great deal of extensive coverage of all sorts of interesting edge cases at both the unit and functional level, and a corresponding culture of maintaining that test suite well when making new changes, it would have been impossible to be anything like as confident of the port as we were.

As part of the porting work, we split out a couple of substantial chunks of the Launchpad codebase that could easily be decoupled from the core: its Mailman integration and its code import worker. Both of these had substantial dependencies with complex requirements for porting to Python 3, and arranging to be able to do these separately on their own schedule was absolutely worth it. Like disentangling balls of wool, any opportunity you can take to make things less tightly-coupled is probably going to make it easier to disentangle the rest. (I can see a tractable way forward to porting the code import worker, so we may well get that done soon. Our Mailman integration will need to be rewritten, though, since it currently depends on the Python-2-only Mailman 2, and Mailman 3 has a different architecture.)

Python lessons

Our database layer was already in pretty good shape for a port, since at least the modern bits of its table modelling interface were already strict about using Unicode for text columns. If you have any kind of pervasive low-level framework like this, then making it be pedantic at you in advance of a Python 3 port will probably incur much less swearing in the long run, as you won’t be trying to deal with quite so many bytes/text issues at the same time as everything else.

Early in our port, we established a standard set of __future__ imports and started incrementally converting files over to them, mainly because we weren’t yet sure what else to do and it seemed likely to be helpful. absolute_import was definitely reasonable (and not often a problem in our code), and print_function was annoying but necessary. In hindsight I’m not sure about unicode_literals, though. For files that only deal with bytes and text it was reasonable enough, but as I mentioned above there were also a number of cases where we needed literals of the language’s native str type, i.e. bytes in Python 2 and text in Python 3: this was particularly noticeable in WSGI contexts, but also cropped up in some other surprising places. We generally either omitted unicode_literals or used six.ensure_str in such cases, but it was definitely a bit awkward and maybe I should have listened more to people telling me it might be a bad idea.

A lot of Launchpad’s early tests used doctest, mainly in the style where you have text files that interleave narrative commentary with examples. The development team later reached consensus that this was best avoided in most cases, but by then there were far too many doctests to conveniently rewrite in some other form. Porting doctests to Python 3 is really annoying. You run into all the little changes in how objects are represented as text (particularly u'...' versus '...', but plenty of other cases as well); you have next to no tools to do anything useful like skipping individual bits of a doctest that don’t apply; using __future__ imports requires the rather obscure approach of adding the relevant names to the doctest’s globals in the relevant DocFileSuite or DocTestSuite; dealing with many exception tracebacks requires something like zope.testing.renormalizing; and whatever code refactoring tools you’re using probably don’t work properly. Basically, don’t have done that. It did all turn out to be tractable for us in the end, and I managed to avoid using much in the way of fragile doctest extensions aside from the aforementioned zope.testing.renormalizing, but it was not an enjoyable experience.

Regressions

I know of nine regressions that reached Launchpad’s production systems as a result of this porting work; of course there were various other regressions caught by CI or in manual testing. (Considering the size of this project, I count it as a resounding success that there were only nine production issues, and that for the most part we were able to fix them quickly.)

Equality testing of removed database objects

One of the things we had to do while porting to Python 3 was to implement the __eq__, __ne__, and __hash__ special methods for all our database objects. This was quite conceptually fiddly, because doing this requires knowing each object’s primary key, and that may not yet be available if we’ve created an object in Python but not yet flushed the actual INSERT statement to the database (most of our primary keys are auto-incrementing sequences). We thus had to take care to flush pending SQL statements in such cases in order to ensure that we know the primary keys.

However, it’s possible to have a problem at the other end of the object lifecycle: that is, a Python object might still be reachable in memory even though the underlying row has been DELETEd from the database. In most cases we don’t keep removed objects around for obvious reasons, but it can happen in caching code, and buildd-manager crashed as a result (in fact while it was still running on Python 2). We had to take extra care to avoid this problem.

Debian imports crashed on non-UTF-8 filenames

Python 2 has some unfortunate behaviour around passing bytes or Unicode strings (depending on the platform) to shutil.rmtree, and the combination of some porting work and a particular source package in Debian that contained a non-UTF-8 file name caused us to run into this. The fix was to ensure that the argument passed to shutil.rmtree is a str regardless of Python version.

We’d actually run into something similar before: it’s a subtle porting gotcha, since it’s quite easy to end up passing Unicode strings to shutil.rmtree if you’re in the process of porting your code to Python 3, and you might easily not notice if the file names in your tests are all encoded using UTF-8.

lazr.restful ETags

We eventually got far enough along that we could switch one of our four appserver machines (we have quite a number of other machines too, but the appservers handle web and API requests) to Python 3 and see what happened. By this point our extensive test suite had shaken out the vast majority of the things that could go wrong, but there was always going to be room for some interesting edge cases.

One of the Ubuntu kernel team reported that they were seeing an increase in 412 Precondition Failed errors in some of their scripts that use our webservice API. These can happen when you’re trying to modify an existing resource: the underlying protocol involves sending an If-Match header with the ETag that the client thinks the resource has, and if this doesn’t match the ETag that the server calculates for the resource then the client has to refresh its copy of the resource and try again. We initially thought that this might be legitimate since it can happen in normal operation if you collide with another client making changes to the same resource, but it soon became clear that something stranger was going on: we were getting inconsistent ETags for the same object even when it was unchanged. Since we’d recently switched a quarter of our appservers to Python 3, that was a natural suspect.

Our lazr.restful package provides the framework for our webservice API, and roughly speaking it generates ETags by serializing objects into some kind of canonical form and hashing the result. Unfortunately the serialization was dependent on the Python version in a few ways, and in particular it serialized lists of strings such as lists of bug tags differently: Python 2 used [u'foo', u'bar', u'baz'] where Python 3 used ['foo', 'bar', 'baz']. In lazr.restful 1.0.3 we switched to using JSON for this, removing the Python version dependency and ensuring consistent behaviour between appservers.

Memory leaks

This problem took the longest to solve. We noticed fairly quickly from our graphs that the appserver machine we’d switched to Python 3 had a serious memory leak. Our appservers had always been a bit leaky, but now it wasn’t so much “a small hole that we can bail occasionally” as “the boat is sinking rapidly”:

(Yes, this got in the way of working out what was going on with ETags for a while.)

I spent ages messing around with various attempts to fix this. Since only a quarter of our appservers were affected, and we could get by on 75% capacity for a while, it wasn’t urgent but it was definitely annoying. After spending some quality time with objgraph, for some time I thought traceback reference cycles might be at fault, and I sent a number of fixes to various upstream projects for those (e.g. zope.pagetemplate). Those didn’t help the leaks much though, and after a while it became clear to me that this couldn’t be the sole problem: Python has a cyclic garbage collector that will eventually collect reference cycles as long as there are no strong references to any objects in them, although it might not happen very quickly. Something else must be going on.

Debugging reference leaks in any non-trivial and long-running Python program is extremely arduous, especially with ORMs that naturally tend to end up with lots of cycles and caches. After a while I formed a hypothesis that zope.server might be keeping a strong reference to something, although I never managed to nail it down more firmly than that. This was an attractive theory as we were already in the process of migrating to Gunicorn for other reasons anyway, and Gunicorn also has a convenient max_requests setting that’s good at mitigating memory leaks. Getting this all in place took some time, but once we did we found that everything was much more stable:

This isn’t completely satisfying as we never quite got to the bottom of the leak itself, and it’s entirely possible that we’ve only papered over it using max_requests: I expect we’ll gradually back off on how frequently we restart workers over time to try to track this down. However, pragmatically, it’s no longer an operational concern.

Mirror prober HTTPS proxy handling

After we switched our script servers to Python 3, we had several reports of mirror probing failures. (Launchpad keeps lists of Ubuntu archive and image mirrors, and probes them every so often to check that they’re reasonably complete and up to date.) This only affected HTTPS mirrors when probed via a proxy server, support for which is a relatively recent feature in Launchpad and involved some code that we never managed to unit-test properly: of course this is exactly the code that went wrong. Sadly I wasn’t able to sort out that gap, but at least the fix was simple.

Non-MIME-encoded email headers

As I mentioned above, there were substantial changes in the email package between Python 2 and 3, and indeed between minor versions of Python 3. Our test coverage here is pretty good, but it’s an area where it’s very easy to have gaps. We noticed that a script that processes incoming email was crashing on messages with headers that were non-ASCII but not MIME-encoded (and indeed then crashing again when it tried to send a notification of the crash!). The only examples of these I looked at were spam, but we still didn’t want to crash on them.

The fix involved being somewhat more careful about both the handling of headers returned by Python’s email parser and the building of outgoing email notifications. This seems to be working well so far, although I wouldn’t be surprised to find the odd other incorrect detail in this sort of area.

Failure to handle non-ISO-8859-1 URL-encoded form input

Remember how I said that parsing HTTP form data was thorny? After we finished upgrading all our appservers to Python 3, people started reporting that they couldn’t post Unicode comments to bugs, which turned out to be only if the attempt was made using JavaScript, and was because I hadn’t quite managed to get URL-encoded form data working properly with zope.publisher and multipart. The current standard describes the URL-encoded format for form data as “in many ways an aberrant monstrosity”, so this was no great surprise.

Part of the problem was some very strange choices in zope.publisher dating back to 2004 or earlier, which I attempted to clean up and simplify. The rest was that Python 2’s urlparse.parse_qs unconditionally decodes percent-encoded sequences as ISO-8859-1 if they’re passed in as part of a Unicode string, so multipart needs to work around this on Python 2.

I’m still not completely confident that this is correct in all situations, but at least now that we’re on Python 3 everywhere the matrix of cases we need to care about is smaller.

Inconsistent marshalling of Loggerhead’s disk cache

We use Loggerhead for providing web browsing of Bazaar branches. When we upgraded one of its two servers to Python 3, we immediately noticed that the one still on Python 2 was failing to read back its revision information cache, which it stores in a database on disk. (We noticed this because it caused a deployment to fail: when we tried to roll out new code to the instance still on Python 2, Nagios checks had already caused an incompatible cache to be written for one branch from the Python 3 instance.)

This turned out to be a similar problem to the pickle issue mentioned above, except this one was with marshal, which I didn’t think to look for because it’s a relatively obscure module mostly used for internal purposes by Python itself; I’m not sure that Loggerhead should really be using it in the first place. The fix was relatively straightforward, complicated mainly by now needing to cope with throwing away unreadable cache data.

Ironically, if we’d just gone ahead and taken the nominally riskier path of upgrading both servers at the same time, we might never have had a problem here.

Intermittent bzr failures

Finally, after we upgraded one of our two Bazaar codehosting servers to Python 3, we had a report of intermittent bzr branch hangs. After some digging I found this in our logs:

Traceback (most recent call last):
  ...
  File "/srv/bazaar.launchpad.net/production/codehosting1-rev-20124175fa98fcb4b43973265a1561174418f4bd/env/lib/python3.5/site-packages/twisted/conch/ssh/channel.py", line 136, in addWindowBytes
    self.startWriting()
  File "/srv/bazaar.launchpad.net/production/codehosting1-rev-20124175fa98fcb4b43973265a1561174418f4bd/env/lib/python3.5/site-packages/lazr/sshserver/session.py", line 88, in startWriting
    resumeProducing()
  File "/srv/bazaar.launchpad.net/production/codehosting1-rev-20124175fa98fcb4b43973265a1561174418f4bd/env/lib/python3.5/site-packages/twisted/internet/process.py", line 894, in resumeProducing
    for p in self.pipes.itervalues():
builtins.AttributeError: 'dict' object has no attribute 'itervalues'

I’d seen this before in our git hosting service: it was a bug in Twisted’s Python 3 port, fixed after 20.3.0 but unfortunately after the last release that supported Python 2, so we had to backport that patch. Using the same backport dealt with this.

Onwards!

Russ Allbery: Review: Piranesi

2 August, 2021 - 11:12

Review: Piranesi, by Susanna Clarke

Publisher: Bloomsbury Publishing Copyright: 2020 ISBN: 1-63557-564-8 Format: Kindle Pages: 245

Piranesi is a story told in first-person journal entries by someone who lives in a three-floored world of endless halls full of statues. The writing style is one of the most distinctive things about this book (and something you'll have to get along with to enjoy it), so it's worth quoting a longer passage from the introductory description of the world:

I am determined to explore as much of the World as I can in my lifetime. To this end I have travelled as far as the Nine-Hundred-and-Sixtieth Hall to the West, the Eight-Hundred-and-Ninetieth Hall to to the North and the Seven-Hundred-and-Sixty-Eighth Hall to the South. I have climbed up to the Upper Halls where Clouds move in slow procession and Statues appear suddenly out of the Mists. I have explored the Drowned Halls where the Dark Waters are carpeted with white water lilies. I have seen the Derelict Halls of the East where Ceilings, Floors — sometimes even Walls! — have collapsed and the dimness is split by shafts of grey Light.

In all these places I have stood in Doorways and looked ahead. I have never seen any indication that the World was coming to an End, but only the regular progression of Halls and Passageways into the Far Distance.

No Hall, no Vestibule, no Staircase, no Passage is without its Statues. In most Halls they cover all the available space, though here and there you will find an Empty Plinth, Niche or Apse, or even a blank space on a Wall otherwise encrusted with Statues. These Absences are as mysterious in their way as the Statues themselves.

So far as the protagonist knows, the world contains only one other living person, the Other, and thirteen dead ones who exist only as bones. The Other is a scientist searching for Great and Secret Knowledge, and calls the protagonist Piranesi, which is odd because that is not the protagonist's name.

Be warned that I'm skating around spoilers for the rest of this review. I don't think I'm giving away anything that would ruin the book, but the nature of the story takes some sharp turns. If knowing anything about that would spoil the book for you and you want to read this without that knowledge, you may want to stop reading here.

I also want to disclose early in this review that I wanted this to be a different book than it is, and that had a significant impact on how much I enjoyed it. Someone who came to it with different expectations may have a different and more enjoyable experience.

I was engrossed by the strange world, the atmosphere, and the mystery of the halls full of statues. The protagonist is also interested in the same things, and the early part of the book is full of discussion of exploration, scientific investigation, and attempts to understand the nature of the world. That led me to hope for the sort of fantasy novel in which the setting is a character and where understanding the setting is a significant part of the plot.

Piranesi is not that book. The story that Clarke wants to tell is centered on psychology rather than setting. The setting does not become a character, nor do we learn much about it by the end of the book. While we do learn how the protagonist came to be in this world, my first thought when that revelation starts halfway through the book was "this is going to be disappointing." And, indeed, it was.

I say all of this because I think Piranesi looks, from both its synopsis and from the first few chapters, like it's going to be a world building and exploration fantasy. I think it runs a high risk of disappointing readers in the way that it disappointed me, and that can lead to disliking a book one may have enjoyed if one had read it in a different mood and with a different set of expectations.

Piranesi is, instead, about how the protagonist constructs the world, about the effect of trauma on that construction, and about the complexities hidden behind the idea of recovery. And there is a lot to like here: The ending is complex and subtle and does not arrive at easy answers (although I also found it very sad), and although Clarke, by the end of the book, is using the setting primarily as metaphor, the descriptions remain vivid and immersive. I still want the book that I thought I was reading, but I want that book in large part because the fragments of that book that are in this one are so compelling and engrossing.

What did not work for me was every character in the book except for the protagonist and one supporting character.

The relationship between the protagonist and the Other early in the book is a lovely bit of unsettling complexity. It's obvious that the Other has a far different outlook on the world than the protagonist, but the protagonist seems unaware of it. It's also obvious that the Other is a bit of a jerk, but I was hoping for a twist that showed additional complexity in his character. Sadly, when we get the twist, it's not in the direction of more complexity. Instead, it leads to a highly irritating plot that is unnecessarily prolonged through the protagonist being gullible and child-like in the face of blatantly obvious gaslighting. This is a pattern for the rest of the book: Once villains appear on stage, they're one-note narcissists with essentially no depth.

There is one character in Piranesi that I liked as well or better than the protagonist, but they only show up late in the story and get very little character development. Clarke sketches the outline of a character I wanted to learn much more about, but never gives us the details on the page. That leads to what I thought was too much telling rather than showing in the protagonist's relationships at the end of the book, which is part of why I thought the ending was so sad. What the protagonist loses is obvious to me (and lines up with the loss I felt when the book didn't turn out to be what I was hoping it would be); what the protagonist gains is less obvious, is working more on the metaphorical level of the story than the literal level, and is more narrated than shown.

In other words, this is psychological fantasy with literary sensibilities told in a frame that looks like exploration fantasy. Parts of it, particularly the descriptions and the sense of place, are quite skillful, but the plot, once revealed, is superficial, obvious, and disappointing. I think it's possible this shift in the reader's sense of what type of book they're reading is intentional on Clarke's part, since it works with the metaphorical topic of the book. But it's not the existence of a shift itself that is my primary objection. I like psychological fantasy as well as exploration fantasy. It's that I thought the book after the shift was shallower, less interesting, and more predictable than the book before the shift.

The one thing that is excellent throughout Piranesi, though, is the mood. It takes a bit to get used to the protagonist's writing style (and I continue to dislike the Affectation of capitalizing Nouns when writing in English), but it's open-hearted, curious, thoughtful, observant, and capable in a way I found delightful. Some of the events in this book are quite dark, but it never felt horrifying or oppressive because the protagonist remains so determinedly optimistic and upbeat, even when yanked around by the world's most obvious and blatant gaslighting. That persistent hopefulness and lightness is a good feature in a book published in 2020 and is what carried me through the parts of the story I didn't care for.

I wish this had been a different book than it was, or failing that, a book with more complex and interesting supporting characters and plot to fit its complex and interesting psychological arc. I also wish that Clarke had done something more interesting with gender in this novel; it felt like she was setting that up for much of the book, and then it never happened. Ah well.

As is, I can't recommend Piranesi, but I can say the protagonist, atmosphere, and sense of place are very well done and I think it will work for some other readers better than it did for me.

Rating: 6 out of 10

François Marier: Time-stretch in Kodi

2 August, 2021 - 01:47

VLC has a really neat feature which consists of time-stretching audio to allow users to speed up or slow video playback with the [ and ] keys without affecting the pitch of the sound. I recently switched to Kodi as my video player of choice and I was looking for the equivalent feature.

Kodi equivalent

To enable this feature in Kodi, you first need to enable Sync playback to display in Settings | Player | Videos.

Then map the tempoup and tempodown commands to the same keyboard shorcuts as VLC.

In my case however, I wanted to map these functions to buttons on my Streamzap remote and so I put the following in my ~/.kodi/userdata/keymaps/remote.xml:

  <FullscreenVideo>
    <remote>
      <pageminus>PlayerControl(tempodown)</pageminus>
      <pageplus>PlayerControl(tempoup)</pageplus>
    </remote>
  </FullscreenVideo>

which allows me to press the Ch + and Ch - buttons on the remote to adjust the speed while the video is playing (in full-screen mode only, not with the menu displayed).

Examples

Here are three ways I use this functionality:

  • I set it to 0.9x for movies in languages I'm not totally proficient in.
  • I set it to 1.1x for almost everything since the difference is not especially perceptible, but it still allows me to watch 10% more movies in the same amount of time
  • I set it to 1.2x for Rick & Morty because it makes Rick even more hilariously reckless and impatient.

Unfortunately, I haven't found a way to set the default tempo value. The closest setting I could find is the one which allows you to set the maximum tempo value maxtempo. If you know of a way, please leave a comment!

Russ Allbery: Review: Fugitive Telemetry

1 August, 2021 - 11:26

Review: Fugitive Telemetry, by Martha Wells

Series: Murderbot Diaries #6 Publisher: Tordotcom Copyright: April 2021 ISBN: 1-250-76538-2 Format: Kindle Pages: 167

Fugitive Telemetry is the fifth Murderbot novella. It is not a sequel to the (as yet) lone novel, Network Effect. Instead, it takes place between Exit Strategy and Network Effect, filling in more of the backstory of the novel. You should not read it before Exit Strategy, but I believe it and Network Effect could be read in any order.

A human has been murdered on Preservation Station. That is not a thing that happens on Preservation Station, which is normally a peaceful place whose crime is limited to intoxication-related stupidity. Murderbot's first worry, and the first worry of his humans, is that this may be one of their enemies getting into position to target them. That risk at least makes the murder worth investigating, rather than leaving it solely to Station Security.

The problem from Murderbot's perspective is that there is an effective and efficient way of doing such an investigation, which starts with hacking into the security systems to get necessary investigative data and may end with the silent disposal of dead bodies of enemy agents. But this is Preservation Station, not the Corporation Rim, and Murderbot agreed to not do things like casually compromise all the station security systems or murder people who are security threats.

There was a big huge deal about it, and Security was all "but what if it take over the station's systems and kills everybody" and Pin-Lee told them "if it wanted to do that it would have done it by now," which in hindsight was probably not the best response.

Worse, Murderbot's human wants it to work collaboratively with Station Security. That is a challenge, given that Security has a lot of reasons not to trust SecUnits, and Murderbot has a lot of reasons not to trust a security organization (not to mention considers them largely incompetent). Also, the surveillance systems are totally inadequate compared to the Corporation Rim for various financial and civil rights reasons that are doubtless wonderful except in situations where someone has been murdered. But hopefully the humans won't get in the way too much.

This is one of those books (well, novellas) that I finished a while back but then stalled out on reviewing. I think that's because I don't have that much to say about it. Network Effect pushed the world-building and Murderbot's personal storyline forward significantly, but Fugitive Telemetry doesn't pick up those threads. Instead, this is another novella in much the same vein as the first four. If you, like me, are eager to see where Wells takes the story after the events of the novel, this is somewhat disappointing. But if you enjoyed the novellas, this is more of what you enjoyed: snarky comments about humanity, competence porn, Murderbot getting pulled into problems somewhat against its will and then trying to sort them out, and the occasional touching moment of emotional connection that Murderbot escapes from as quickly as possible.

It's quite enjoyable, helped considerably by Wells's wise choice to not make the supporting human characters idiots. Collaboration is not Murderbot's strength; it is certain the investigation will be an endless series of frustrations and annoyances given the level of suspicion Station Security starts with. But some humans (and some SecUnits) are capable of re-evaluating their conclusions when given new evidence, and watching that happen is part of the fun of this novella.

What this novella is missing is the overarching plot structure of the rest of the series, since where this story sits chronologically doesn't leave much room for advancing or even deepening the plot arc. It therefore feels incidental: delightful while I was reading it, probably missable if you have to, and not something I spent time thinking about after I finished it.

If you liked the Murderbot novellas up until now, you will want to read this one. If you haven't started the series yet, this is not a place to start. If you want something more like the Network Effect novel, or a story where Murderbot makes significant decisions about its future, the wait continues.

Rating: 8 out of 10

Paul Wise: FLOSS Activities July 2021

1 August, 2021 - 08:54
Focus

This month I didn't have any particular focus. I just worked on issues in my info bubble.

Changes Issues Review Administration
  • libusbgx/gt: triage issues
  • Debian packages: triaged bugs for reintroduced packages
  • Debian servers: debug lists mail issue, debug lists subscription issue
  • Debian wiki: unblock IP addresses, approve accounts
Communication
  • Respond to queries from Debian users and contributors on the mailing lists and IRC
Sponsors

The microsoft-authentication-library-for-python and purple-discord work was sponsored by my employer. All other work was done on a volunteer basis.

Junichi Uekawa: August comes.

1 August, 2021 - 08:29
August comes. Kids are on summer staycation. This is not sustainable.

Steinar H. Gunderson: How to optimize anything

1 August, 2021 - 06:00

Speeding up software, in four simple, universal steps:

  1. Make a benchmark.
  2. Run a profiler over that benchmark.
  3. Try something reasonable (based on #2) to speed up the benchmark.
  4. If the benchmark gets faster, clean the code up and commit.

Repeat steps 2–4 until the code is fast enough.

Of course, most people stumble in step 1 (e.g. by making a benchmark that is non-representative, like PHP 8's infamous JIT that helped 3x on the benchmark, but at most 3–5% on real code). And step 3 is naturally where all the magic happens. The cheapest wins often come out of a surprising profile, and the best wins often come from taking a step up and optimizing at a higher level. The most satisfying ideas are those that simplify code, so that you end up with just running less stuff and having things look more natural. (The worst ideas come when you skip step 2, because you end up optimizing what you think takes time, which is rarely the stuff that actually does.)

The “something reasonable” part is mandatory, or you are likely to just measure incidental effects. ryg lays down the law.

Chris Lamb: Free software activities in July 2021

31 July, 2021 - 23:08

Here is my monthly update covering what I have been doing in the free software world during July 2021 (previous month):


SPI is a non-profit corporation that acts as a fiscal sponsor for organisations that develop open source software and hardware.

As part of my role of being the assistant Secretary of the Open Source Initiative and a board director of Software in the Public Interest I attended their respective monthly meetings. As outlined in last months posts, however, my term on the OSI board has been slightly extended due to the discovery of a vulnerability in OSI's recent election — as a result, the 2021 election is currently being re-run.

§

Reproducible Builds

One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes. The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

This month, I:

  • Updated the Lintian static analysis tool to check for Python tracebacks in manual pages, usually caused by failing help2man calls and the cause of avoidable reproducibility issues. (#984778 filed against the heudiconv package is a good example of the problem.) [...]


diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues.

I also made the following changes to diffoscope, including preparing and uploading versions 178 and 179 to PyPI and Debian:

  • Ensure that various LLVM tools are installed, even when testing whether a MacOS binary has no differences compared to itself. (#270)
  • Rewrite how we calculate the 'fuzzy hash' of a file to make the control flow cleaner. [...][...]
  • Don't traceback when encountering a broken symlink within a directory. (#269)
  • Update some copyright years. [...]

§


Debian Bugs filed Uploads
  • redis:

    • 6.0.15-1 — New upstream security release.
    • 6.2.5-1 (to Debian experimental) — New upstream security release.
  • python-django:

    • 3.2.5-1 (to Debian experimental) — New upstream security release.
    • 3.2.5-2 (to Debian experimental) — Don't symlink /usr/bin/django-admin to django-admin.py. Instead, ship the script generated by the Python entry_points system, otherwise we introduce a confusing django-admin.py-related deprecation message when using django-admin (ie. without the .py extension). (#991098)
  • mtools:

    • 4.0.32-1 — New upstream release.
    • 4.0.33-1 — New upstream release.
    • 4.0.33-1+really4.0.32-1 — Revert to version 4.0.32-1 due to regressions on ARM systems affecting the Debian Installer. (#991403)

§

Debian LTS

This month I have worked 18 hours on Debian Long Term Support (LTS) and 12 hours on its sister Extended LTS project.

You can find out more about the project via the following video:

Jamie McClelland: Fixing old PHP code

31 July, 2021 - 22:11

I wrote a control panel in 2005 using PHP, without any framework. Who could have guessed it would still be in production now?

We’ve recently decided to put off replacing it for a few years, so I have to fix all the deprecation warnings, which are almost all due to:

while(list($k, $v) = each($array)) {

At some point, early in my PHP coding life, someone told me foreach($array as $k => $v) { was bad. I don’t even remember why. But it stuck, so my code is littered with the while/list/each approach. If I ever wrote malware in PHP you could definitely fingerprint me with this one.

I’m working on some sed magic to fix them, starting with:

find . -name '*.php' -exec sed -E -i 's#while\(list\((\$[a-z_]*), ?(\$[a-z_]*)\) = each\((\$[a-z_>-]+)\)\) \{#foreach(\3 as \1 => \2) {#g' '{}' \;

But… it misses this variation:

while(list(, $v) = each($array)) {

So I also ran:

find . -name '*.php' -exec sed -E -i 's#while\(list\(,(\$[a-z_]*)\) = each\((\$[a-z_>-]+)\)\) \{#foreach(\2 as \1) {#g' '{}' \;

I ended up with 10 replacments I had to do by hand (while(list($k) = each($array)) and a few others with unusual spacing).

Russell Coker: Links July 2021

31 July, 2021 - 20:16

The News Tribune published an article in 2004 about the “Dove of Oneness”, a mentally ill woman who got thousands of people to believe her crazy ideas about NESARA [1]. In recent time the QANON conspiracy theory has drawn on the NESARA cult and encouraged it’s believers to borrow money and spend it in the belief that all debts will be forgiven (something which was not part of NESARA). The Wikipedia page about NESARA (proposed US legislation that was never considered by the US congress) notes that the second edition of the book about it was titled “Draining the Swamp: The NESARA Story – Monetary and Fiscal Policy Reform“. It seems like the Trump cult has been following that for a long time.

David Brin (best-selling SciFi Author and NASA consultant) wrote an insightful blog post about the “Tytler Calumny” [2], which is the false claim that democracy inevitably fails because poor people vote themselves money. When really the failure is of corrupt rich people subverting the government processes to enrich themselves at the expense of their country. It’s worth reading, and his entire blog is also worth reading.

Cory Doctorow has an insightful article about his own battle with tobacco addiction and the methods that tobacco companies and other horrible organisations use to prevent honest discussion about legislation [3].

Cory Doctorow has an insightful article about “consent theater” which is describes how “consent” in most agreements between corporations and people is a fraud [4]. The new GDPR sounds good.

The forum for the War Thunder game had a discussion on the accuracy of the Challenger 2 tank which ended up with a man who claims to be a UK tank commander posting part of a classified repair manual [5]. That’s pretty amusing, and also good advertising for War Thunder. After reading about this I discovered that it’s free on Steam and runs on Linux! Unfortunately it whinged about my video drivers and refused to run.

Corey Doctorow has an insightful and well researched article about the way the housing market works in the US [6]. For house prices to increase conditions for renters need to be worse, that may work for home owners in the short term but then in the long term their children and grandchildren will end up renting.

Related posts:

  1. Links April 2021 Dr Justin Lehmiller’s blog post comparing his official (academic style)...
  2. Links July 2020 iMore has an insightful article about Apple’s transition to the...
  3. Links February 2021 Elestic Search gets a new license to deal with AWS...

Russ Allbery: Summer haul

31 July, 2021 - 12:37

July ended up being a very busy month for me catching up on all sorts of things that I'd been putting off for too long, so posts have been a bit scarce recently. So have book reviews; I'm hoping to sneak one in before the end of the month tomorrow, and have a small backlog.

But for tonight, here's another list of random books, mostly new releases, that caught my eye.

Katherine Addison — The Witness for the Dead (sff)
Olivia Atwater — Half a Soul (sff)
Lloyd Biggle, Jr. — The Still, Small Voice of Trumpets (sff)
Judson Brewer — Unwinding Anxiety (nonfiction)
Eliot Brown & Maureen Farrell — The Cult of We (nonfiction)
Becky Chambers — A Psalm for the Wild-Built (sff)
Susanna Clarke — Piranesi (sff)
Eve L. Ewing — Ghosts in the Schoolyard (nonfiction)
Michael Lewis — The Premonition (nonfiction)
Courtney Milan — The Duke Who Didn't (romance)
Kit Rocha — Deal with the Devil (sff)
Tasha Suri — The Jasmine Throne (sff)
Catherynne M. Valente — The Past is Red (sff)

Quite a variety of things recently. Of course, I'm currently stalled on a book I'm not enjoying very much (but want to finish anyway since I like reviewing all award nominees).

Dirk Eddelbuettel: RcppAnnoy 0.0.19 on CRAN: Maintenance

31 July, 2021 - 09:17

A minor maintenance release, now at version 0.0.19, of RcppAnnoy is now on CRAN. RcppAnnoy is the Rcpp-based R integration of the nifty Annoy library by Erik Bernhardsson. Annoy is a small and lightweight C++ template header library for very fast approximate nearest neighbours—originally developed to drive the famous Spotify music discovery algorithm.

This release only contains internal packaging changes. Nothing changes upstream, or in package functionality. Detailed changes follow.

Changes in version 0.0.19 (2021-07-30)
  • Minor tweaks to default CI setup and DESCRIPTION file

Courtesy of my CRANberries, there is also a diffstat report for this release.

If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Jonathan Dowland: Accounting: pooling income

30 July, 2021 - 22:44

I wrote about budgeting nine years ago and I've been a little reluctant to write about it again: by far, it's the blog post that has attracted the most requests from people asking me to link to their blog, site, or service.

I wasn't good at budgeting then and I'm still not good at it now, although I have learned a few things in the intervening time. Those things more properly relate to accounting than budgeting (so there's the first thing: I learned the difference!). I wanted to write about some of the things I've learned since then, starting with our family's approach to pooling income.

Pooling

From talking to friends about how they manage stuff, this doesn't seem to be a common approach. We pay all our income into a shared account. We agree on an amount of "play money" that we can individually spend on whatever we like, and we pay that amount to ourselves from the shared account every month. Crucially, the amount we pick is the same for each of us, irrespective of our relative incomes. All of our shared family expenses come out of the shared account.

Some of my friends, especially (exclusively) the bread-winners, find this a bit alarming. One of the things I like about it is that whichever partner earns less than the other is not disadvantaged in terms of their discretionary spending. When my wife earned less than me, and I believe structural sexism was a contributing factor to that, that impacted us both equally. When my wife was not earning a salary at all, but was doing the lion's share of bringing up our children, she has the same discretionary spend as I do. Apart from the equity of it, there's a whole class of gripes and grumbles that some of my friends have about their partner's spending habits or money management that we completely avoid.

Anton Gladky: 2021/07, FLOSS activity

30 July, 2021 - 21:00
LTS

This is my fifth month of working for LTS. I was assigned 12 hrs and worked all of them.

Released DLAs
  1. DLA 2705-1 scilab_5.5.2-4+deb9u1

    • CVE-2021-31598: Out-of-bounds write in ezxml_decode() leading to heap corruption
    • CVE-2021-31347, CVE-2021-31348: incorrect memory handling in ezxml_parse_str() leading to out-of-bounds read
    • CVE-2021-31229: Out-of-bounds write in ezxml_internal_dtd() leading to out-of-bounds write of a one byte constant
    • CVE-2021-30485: incorrect memory handling, leading to a NULL pointer dereference in ezxml_internal_dtd()

    With this upload not all opened CVEs were closed in this package. Because some of CVEs were not fixed yet by upstream. Added links to upstream bug reports for the following CVEs: CVE-2021-31598 CVE-2021-31348 CVE-2021-31347 CVE-2021-31229 CVE-2021-30485 CVE-2021-26222 CVE-2021-26221 CVE-2021-26220 CVE-2019-20202 CVE-2019-20201 CVE-2019-20200 CVE-2019-20199 CVE-2019-20198 CVE-2019-20007 CVE-2019-20006 CVE-2019-20005 into the data/CVE/list on securoty tracker.

  2. DLA 2707-1 sogo_3.2.6-2+deb9u1

    • CVE-2021-33054: SOGo does not validate the signatures of any SAML assertions it receives. Any actor with network access to the deployment could impersonate users when SAML is the authentication method.
Other LTS-related work LTS-Meeting

I attended the Debian LTS team IRC-meeting this month.

Other FLOSS activities
  1. One week before the full freeze of Debian Bullseye the release-critical bug #990895 against the package httraqt was filed. Thanks to the reporter I could fix it within the hour after the ticket was created, uploaded as the version httraqt_1.4.9-5, filed an unblock-request, which was approved.

Reproducible Builds (diffoscope): diffoscope 179 released

30 July, 2021 - 07:00

The diffoscope maintainers are pleased to announce the release of diffoscope version 179. This version includes the following changes:

* Ensure that various LLVM tools are installed, even when testing whether
  a MacOS binary has zero differences when compared to itself.
  (Closes: reproducible-builds/diffoscope#270)

You find out more by visiting the project homepage.

Patryk Cisek: Debian on TrueNAS Core under bhyve

29 July, 2021 - 05:45
Installing Debian/GNU Linux under bhyve on TrueNAS Core I got myself a TrueNAS Mini X+ couple of months ago. I have it running TrueNAS Core based on FreeBSD. In that system you can run VMs under FreeBSD’s native hypervisor, bhyve. Since there are a couple of quirks around running Debian specifically, I decided to write up a quick article about setting up Debian-based VM there. The quirks The ones I’ve stumbled upon were:

Pages

Creative Commons License ลิขสิทธิ์ของบทความเป็นของเจ้าของบทความแต่ละชิ้น
ผลงานนี้ ใช้สัญญาอนุญาตของครีเอทีฟคอมมอนส์แบบ แสดงที่มา-อนุญาตแบบเดียวกัน 3.0 ที่ยังไม่ได้ปรับแก้