Bonum Certa Men Certa

The Latest GitHub-Free Research: An Introduction or Update

Article by figosdev

The gearscape
Microsoft interjects itself as dependency to control the competition

Summary: An examination of how dependent on Microsoft's proprietary jail (GitHub) GNU/Linux distros have become

When I started to find old favourites were hosted on GitHub, I became increasingly interested in what was there. The more I looked, the more I found. I've tried to get a better idea of the scope of the problem ever since.



Automation (scripting, mostly) can and does assist me in this, but there is no automated way to determine if a project is GitHub-based or not. I'll check DistroWatch, Wikipedia, frequently the project's own website, but some of the discoveries require going into the file -- particularly when that file is an ISO image.

"The goal is not to get perfect data (simply impossible with the number of people involved) but to get finer resolution and hopefully gain accuracy with each pass and research stage."I started with a list that reached about 100 to 120 popular applications and distros that are based on GitHub. It was a first effort, also the most casual, and when I "audited" the list later on, I found it "relying on GitHub" if someone happens to do development and issues there, and relying on it for hosting are all a problem in my opinion. That's what I'm looking for -- projects that can actually be hurt if Microsoft either pulls the plug or reduces access.

I'm certainly not impressed by their recent invitation to use formerly premium features. GitHub is a trap, and now they're opening the trap a little wider.

This is an imperfect science, but the more information we have, the better. Some of the research I'm doing builds on what I know, but some of the research is redundant by design, and acts as a check on previous research. The goal is not to get perfect data (simply impossible with the number of people involved) but to get finer resolution and hopefully gain accuracy with each pass and research stage.

"When I was ranking levels of distro dependency on GitHub, naturally I put actually being hosted and developed on GitHub as a key problem, if not the worst."The big picture matters as much as the details. Syslinux is more or less necessary in my opinion, and is not GitHub-based. But I found out while writing this that syslinux uses mkdiskimage -- a Perl script. Am I going to count this as needing GitHub? Probably not, but I'm going to make note of it here.

It is necessary, during each stage of this research, to choose a methodology. When I was ranking levels of distro dependency on GitHub, naturally I put actually being hosted and developed on GitHub as a key problem, if not the worst.

I'm trying to build a practical, useful idea of where the hope is, and where it isn't -- and each stage is practical in the sense that previous research sheds light on what I'm doing now, and vice-versa. Looking for applications helped me rank dependent distros. Ranking distro dependency helped me find new (and more vital) applications and frameworks. This is the best kind of research, where nearly every bit of it helps in some way.

"Tiny Core was one of the 33 most GitHub-free distros, of the 275 examined in my previous research."Ideally, we want all of the bad news we can find, and all of the good news we can find. As for how it's done, every methodology has pros and cons. Right now I'm trying to find the best way to examine Tiny Core Linux, and I've spent days looking at CorePlus in particular -- because I'm also trying to migrate to a less GitHub-dependent distro.

Tiny Core was one of the 33 most GitHub-free distros, of the 275 examined in my previous research. It's also one I'm familiar with; I was one of the early Tiny Core users after having purchased the book Shingledecker and Andrews did. Their disagreement over how to do things led to Tiny Core, and I think it's one of the better (beneficial) separations and forks that have happened in Free software. It's also fun to take apart and explore.

I discovered (or rediscovered, after years of being away) that TC uses .info files to map package dependencies. If you've ever written a recursive function, even if you don't do it all the time, it's very helpful for this sort of thing (I sometimes teach beginner-level coding, and recursion comes up but not a lot.)

This hastily-constructed python function let me recursively create lists of packages that needed packages that needed packages that needed for example, libffi.tcz:

#### license: creative commons cc0 1.0 (public domain) 
#### http://creativecommons.org/publicdomain/zero/1.0/ 
which = sys.argv[1:]
if not which: which = ["libffi.tcz"]



def wget(ls): copy = [] for which in ls: now = "ls *.dep" for each in figarrshell(now): if each: pf = open(each) p = pf.read().replace(chr(13) + chr(10), chr(10)).replace(chr(13), chr(10)).split(chr(10)) ; pf.close() for ln in p: if ln == which and len(ln) > 0: pr = each.replace(".dep", "") if pr not in copy and pr not in ls: copy += [pr] ; print pr if copy: wget(copy)

wget(which)



I also found that open().read() works slightly differently in CPython (GitHub) and PyPy (foss.heptapod.net) although figarropen() does work with PyPy as a drop-in replacement for Python 2. What's different is the limit of open files you can have; I always assumed (without evidence to the contrary) that open().read() closes the file (as does "with" in Python.)

In PyPy, it got to about 1,200 open files (I was opening 2,321 files but assumed they would be closed when read) before complaining that too many were open. I simply opened the file separately from the read() method, so I could run close() after read(). This would be a good change to add to fig as well, so it works better with PyPy. Even fig (most recent update, 2017) stands to benefit from this research.

But the methodology I'm using is to look at the 2,321 packages in TC 11.x, of which I've already dealt with 1,200 -- and find out in what ways Tiny Core "needs" GitHub, or what could be removed to make it less dependent.

I think we could strip out most of the dependencies -- not likely all -- but which ones? And where are they? That's the goal of this stage of research, to find out what is what and which is which.

"The fact that the most-needed "optional" package is based on GitHub (since May 2016) is quite relevant -- 739 (39%) of the 2,321 TC packages need it!"I've already got a list of how many packages need each package, ranked from most to least. Tiny Core is 16mb, CorePlus is 206mb. I know the biggest difference; CorePlus includes a lot more packages. But I've also used Tiny Core more than CorePlus, I know a lot of its limitations, and I'm largely focused on the packages themselves.

Most or all of these packages are technically "optional" (many of them need several others, but that's more or less so with packages in practically every distro) and of course, for example if you have a gtk3 app, it needs gtk3 of reasons that should be obvious. Be assured that I am no fan of doing dependencies to excess, but some dependencies have to be considered reasonable.

I don't know enough about libffi to judge its technical merits or lack thereof. I discovered it through this research (I write scripts mostly, not C or C++ and I don't use ctypes, though I know what it is) because it came up as the "most-needed optional package" in all of this. The fact that the most-needed "optional" package is based on GitHub (since May 2016) is quite relevant -- 739 (39%) of the 2,321 TC packages need it!

Of those 739 libffi-needing packages, 712 of them need glib2 as well (that's GNOME lib, not GNU libc) and ALL of the glib2-based packages need libffi, so perhaps we have GNOME to thank once again. As much as I love blaming them for things, I am unconvinced they're to really blame this time, though the fact remains that if you were to get rid of glib2 then you would also get rid of all but 27 libffi-based packages. You would also get rid of gtk I think, but I'm not suggesting we do that. We build the practical decisions on top of the hard data -- I'm more immediately interested in the data, but the time for decision-making will come.

"Libffi is GitHub, glib2 is Gitlab-based (at any rate, not GitHub) though glib2 does pull in libffi."These are far from isolated examples, indeed the currently methodology is being built on examples like this, which makes them an ideal illustration. Libffi is GitHub, glib2 is Gitlab-based (at any rate, not GitHub) though glib2 does pull in libffi.

We want to show:

* As many of these relationships as possible * With some method of ranking priority, so we know what to check * In some way that shows possible outcomes based on various possible decisions

So the imperfect methodology used here tries to do all of that.

First, I created a folder called libffi. Libffi is GitHub so if we want to get rid of GitHub entirely, we would have to get rid of every package that needs libffi. I don't expect that to happen, I do want to provide as much information on that as necessary so people can at least determine / rate / track / plan / troubleshoot how practical it is for Free software to be independent of a monopolistic company that has always wanted to enslave, tax, control and own it.

I moved all 739 (39%) of the TC packages which need libffi.tcz to the libffi folder. This seems almost too simple to be useful, but we keep going on our mission of discovery.

The next step was to create a folder for glib2. It's not because glib2 depends on libffi, I didn't know that. I learned that because glib2.tcz is the next on the list (712 packages need it) and none of them were in the pool -- they were all in the libffi folder. Now we're learning something.

"The numbers alone do not excite me, though the picture I'm trying to uncover, quantify and find the "shape" of, is of interest."So instead of creating a glib2 folder, I created a libffi/glib2 folder. Most (in fact, all) of the glib2-based packages need libffi, so I just put them under libffi -- but now we know which libffi packages do need glib2 and which don't. Our little hierarchy shows all of that data for us to base decisions on.

And I'm well aware that this system isn't going to stay perfect or neat as it moves forward, it isn't designed to be. My only goal here is to outdo (build on) the previous GitHub-free research, so that new strategies can be gleaned from this. I learned plenty from examining 275 distros, now I want to examine this one in detail. It just happens to be very good for the purpose -- this will tell us about a lot more than just Tiny Core. It already tells us something about libffi and (most likely) about gtk applications. Some of these things will turn out to be specific to Tiny Core. Many will apply broadly. That could be another research project.

Moving forward, I'll share the current hierarchy as it is created. It's not perfectly self-documenting, so I'll document what's there so far (this part is the "introduction" referred to in the title.)

core/libffi: the packages that need libffi.tcz

core/libffi/gitlab: major (next/high on the list) gitlab-based rather than GitHub-based packages that need libffi.tcz

core/libffi/gitlab/glib2: glib2 is gitlab-based; the packages in this folder need both glib2.tcz and libffi.tcz

---

core/selfhost: (next/high on the list) self-hosted (not GitHub or gitlab) packages

core/selfhost/liblzma: liblzma.tcz is next on the list, self-hosted (part of xz utils) and here are the packages that need it.

"People who do databases might correctly assume that an RDB would be better way to organise this data."The list of course, is incomplete -- that's acceptable for this methodology. We already have a full list of required packages for each package; that's how we know there are 689 packages that need liblzma (making it #3 on the list, after glib2) -- we counted them!

For the purpose of this research, it's (only sometimes arbitrarily) more relevant that a package needs a more-needed package than a less-needed package. So instead of the 689 packages we already have a list of, the liblzma sub-hierarchy simply holds the ones "remaining" after "larger" hierarchies are counted. Including subfolders, core/selfhost/liblzma contains 97 packages; the others are needed also by "bigger" "more important" (debatable, hence "sometimes arbitrary") packages that we already sorted.

We could simply get all the data on every package. By "we", I mean people I don't know who haven't signed up to do anything about this. Since I'm doing this, I'm using this imperfect system to discover new areas of focus -- I probably can't make myself do detailed research on EVERY one of the 2,321 packages, so I'm using this system to discover the "most important" problems, based on a methodology that has both pros and cons.

I wouldn't bother with this if I didn't think this approach would shed additional light on the bigger picture. The numbers alone do not excite me, though the picture I'm trying to uncover, quantify and find the "shape" of, is of interest.

"As of this writing, GitHub is only being used for squashfs-tools that create the file systems; the development of squashfs support for the kernel is still on kernel.org."Since this methodology "reveals" that liblzma is important -- and I've learned other things too, this method helps me decide where to pay more attention: I learned that liblzma is part of xz utils, which I didn't know; and I didn't know that xz utils was started by Slackware enthusiasts -- which is both cool, and maybe says something else (something fundamental) about liblzma. You decide what it means to you. This hierarchical system serves as a score -- and for me, a curriculum.

Which leads to the next concept -- noting projects that use GitHub within the hierarchy (where I can spot them or find new ones.)

core/selfhost/liblzma/github: are projects based on GitHub that need liblzma.tcz. It's liblzma that is self-hosted, not the projects in this subfolder; otherwise this path would contradict itself.

People who do databases might correctly assume that an RDB would be better way to organise this data. They would probably be technically correct (I don't normally use databases) but they would be missing the point that we don't actually have the data yet -- the only reason core/selfhost/liblzma/github exists is because core/selfhost/liblzma "told me" to take some time to look for GitHub-based things that needed liblzma.

Will I find them all now? Probably not, but this system informed me to take time on this. We keep trying to shape the data based on relevance, not unlike the earliest versions of the (once fairly straightforward, but also easily-gamed if you want website prominence) PageRank algorithm.

"I figured that if I wanted to focus on fixing this GitHub dependency, I could simply make a live distro that uses a file img formatted with ext3 instead."Incidentally, core/selfhost/liblzma/github includes squashfs-tools (a feature very important to Tiny Core and also to most Live distros, as noted in my previous article) but because it came up again here, I had another look. As of this writing, GitHub is only being used for squashfs-tools that create the file systems; the development of squashfs support for the kernel is still on kernel.org.

In practical terms, this suggests a hypothetical, completely GitHub-free distro could be used to create a tool that reads and converts .sfs files to some other compressed filesystem, though a GitHub-free tool could not (at this time) be made to produce new .sfs images -- only convert them to something else.

I figured that if I wanted to focus on fixing this GitHub dependency, I could simply make a live distro that uses a file img formatted with ext3 instead. How to do compression on the fly could be a separate issue, but I know that alternatives exist.

core/selfhost/ncursesw: means that ncursesw is self-hosted, and this is where the packages that need ncurses.tcz go. Originally there were 665 of them, though now we have 156 remaining due to "more important" packages grabbing those in the hierarchy.

core/selfhost/ncursesw/github: packages moved from ../ to ./ which are based on GitHub, or 25 of 156 packages including: python.tcz (CPython, GitHub), urwid.tcz, tmux (sorry Roy,) inxi.tcz, htop.tcz, freebasic.tcz and vim.tcz.

I think Vim is one of those few things where I'm never sure to say whether it's really relying on GitHub or not. Since I'd hate for Microsoft to end the editor wars with a cure far worse than the disease, I hope someone can give me some truly authoritative evidence that Vim is in fact, GitHub-free. Another thing I found out while doing this is that the person who maintains ncurses is maintained by the same person who maintains the Lynx browser -- and Vile. Vile appears to be GitHub-free, but this research will help determine the validity of that statement. (Vile is not packaged in TC 11.x.)

core/github: was created at this point, for the 43 packages that are actually listed in the .info file as being GitHub-based, minus at least one (pax-utils.tcz, as Gentoo's GitHub is a mirror.)

"With over 1,200 files in these folders, more than 50% of the packages in TC are now sorted into the hierarchy."core/selfhost/libXau-and-libXdmcp: is related to X and these two packages had identical lists, except for libXau-dev.tcz and libXdmcp-dev.tcz, respectively.

core/selfhost/libXau-and-libXdmcp/github contains 12 packages, including wbar.tcz, i3.tcz, aterm.tcz (AfterStep is GitHub-based) and fltk-1.3.tcz.

core/libxcb: was created, and should probably be moved to selfhost, though it has no packages anyway because libxcb-dev.tcz is already in core/selfhost/libXau-and-libXdmcp.

core/libX11: was created and should probably be moved to selfhost, though there's nothing in it.

core/bzip2-lib: has 48 files, including a bunch of Perl-related packages in core/bzip2-lib/github -- Perl is GitHub-based.

With over 1,200 files in these folders, more than 50% of the packages in TC are now sorted into the hierarchy. Some remain undiscovered ties to GitHub, though this process has helped find and rank new ones that are obviously important in some way.

I'm still interested in moving further down the list; the next is libXext.tcz and there are 585 packages that need it. If we try to discover how many of those 585 packages remain...

for p in $(cat ../libXext.tcz.dp) ; do ls ../$p 2> /dev/null ; done | cat -n

...Nope. Nothing there that isn't already in the hierarchy. libXext.tcz.dep is the file that TC provides that shows a single level of dependency, libXext.tcz.dp is the file that the Python code in this article created for libXext.tcz, which shows all the packages that need it.

"Days into this, we've confirmed that TinyCore is indeed one of the least-GitHub-dependent distros, but we've also identified some the more important ways in which it is still dependent indirectly on GitHub."We can use this to create a graph of diminishing returns on this research. Days into this, we've confirmed that TinyCore is indeed one of the least-GitHub-dependent distros, but we've also identified some the more important ways in which it is still dependent indirectly on GitHub.

I thought about making that graph, but since it's likely to be typical and not reveal anything that isn't obvious, I'm just going to watch a movie, eat some eggs and maybe think about getting back to this research. I'm sure it sounds terribly boring, but I continue to learn more about this subject as I explore it.

When this started, I hadn't even thought to start with the most needed packages -- the first thing I wanted to know how many packages pulled in mono.tcz or Perl or Python. Mono is not only GitHub-based, it's one of the worst dependencies you can have. Fortunately, the only packages that pull in mono.tcz are gtk-sharp-dev.tcz, gtk-sharp.tcz, mono-dev.tcz and mono-locale.tcz. I'm only guessing that wine-mono.tcz assumes mono.tcz is installed.

If you're trying to figure out how we can be GitHub-free in the future, I can probably save you some work -- and if you have information that could be useful, by all means, let us know. With luck, this is going to help round out the wiki pages a bit as well.

Long live Stallman, and happy hacking.

Licence: Creative Commons CC0 1.0 (public domain)

Recent Techrights' Posts

"Security Advantages" Explained by a Scammy "Security" Site That Uses LLMs to Spew Out Garbage
destroying the Web by saturating it with "bullshit".
Over at Tux Machines...
GNU/Linux news for the past day
 
Links 12/10/2024: More Site Blocking, China's Hostility, and Evan Gershkovich's Upcoming Book
Links for the day
Links 12/10/2024: Boeing to Cut 17,000 Jobs, Medieval Sleeping Habits, Warning About Liquidweb
Links for the day
Links 12/10/2024: Health, Safety and Climate Concerns
Links for the day
Gemini Links 12/10/2024: Ensemble and Assembler
Links for the day
Links 12/10/2024: TikTok Layoffs and Risk of More Wars
Links for the day
IRC Proceedings: Friday, October 11, 2024
IRC logs for Friday, October 11, 2024
Gemini Links 11/10/2024: Against Cynicism, on Atheism, and Dropping Off The Internet
Links for the day
IBM Employees Smell Another Wave of Mass Layoffs (and Explain the Signs)
IBM currently has the policy of hiding the layoffs from shareholders and from the press using NDAs
Links 11/10/2024: Lots More Censorship and Growing Concerns About Health Impact of Social Control Media
Links for the day
Going Almost 4.5 Decades Back to Find 'Dirt' on a Person
That incident was 42.5 years ago. Is that how far some people would go in an effort to discredit a person?
XBox is Dead. This is Just the Beginning.
the main reason Microsoft bought Activision/Blizzard was to hide the growing losses and failure of XBox
The Risk to the "Linux" Brand
Brands that are not guarded from misuse/abuse will inevitably lose their original meaning and their value
Gemini Links 11/10/2024: Deploying Common Lisp Programs and Examining FreeBSD
Links for the day
Links 11/10/2024: Discord Still Blocked in Turkey, Google Might be Split
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Thursday, October 10, 2024
IRC logs for Thursday, October 10, 2024
LinuxSecurity (Guardian Digital, Inc) Sloppy With Its 'Linux' Slop
This kind of stuff is killing the World Wide Web and ruins human knowledge
[Meme] Chin-dropping and Jaw-dropping (Considerable Drop in Patent Validity and Quality)
This drop is very much intentional
Gemini Links 10/10/2024: Untruth, SSH, Gopher, and More
Links for the day
Geminispace Beyond 4,100 Capsules
4,000 was less than 8 weeks ago
Links 10/10/2024: TikTok's Legal Problems, WeblogPoMo Challenges
Links for the day
[Meme] European Patent Convention and Vienna Convention Became Only Fictions (Laws and Constitutions Are Now Works of Fiction in Europe)
A political crisis and blunder
Almost a Thousand EPO Staff Protesting to EPO Member States That the Office Illegally Grants Software Patents and Other Invalid European Patents
"The outcome confirms that the concerns about the EPO’s ability to grant legally sound patents remain"
Loss of Technical Merit(ocracy)
"buzzword diplomas"
Junk Science
science is being compromised for business purposes
[Meme] Dismantling .io (Stick a Fork, the Hype is Done)
NVIDIA is an excellent new example of hype driving up fictional "value"
UNIX is 55 This Year, It is 6 Years Older Than Microsoft
It should be noted that the surviving co-creator of UNIX, Ken Thompson, 'moved' to GNU/Linux (Debian) in recent years
This Year, for the First Time Since August 2019 (Bill Gates MIT Scandal, Jeffrey Epstein Bribes), libreplanet-discuss Was Inactive an Entire Month
The MIT injustice remains and recent "libreplanet" events were held in a venue that's not MIT and far less prestigious than MIT (the "Wentworth" imitation)
[Meme] Different Ending for Jurassic Park
UNIX in old movies
Evolution of Hype
Passing fads and rebranding
Groklaw Will Hopefully Come Back
Sites should be able to run for decades with hardly any human role/interaction, but that's not where we are...
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Wednesday, October 09, 2024
IRC logs for Wednesday, October 09, 2024