11.28.20

Help Make Techrights (and Other Technology-Centric Sites) More Robust to Censorship by Setting Up More IPFS Nodes

Posted in Site News at 7:03 pm by Dr. Roy Schestowitz

IPFS large logo

Summary: We’re trying to improve the site’s availability (ensuring it can never be offline) and make it more censorship-resistant; people who adopt IPFS can make that happen while tackling the “bloated Web” and “centralised Internet” issues — all at the same time

JUST over a month ago we hooked up with data-centric security folks who had offered help with Techrights. Having faced legal threats over the years (for exposing corruption), we’re always looking for ways to avert or discourage such attempts (ideally, even if we’re forced to go offline or remove something, we want that to be outside our control, i.e. inability to comply, even if there’s a will to comply). We don’t typically just repeat what other sites say, we have many exclusive stories and we publish leaks (unseen beforehand).

Now, less than a month later, we have a pipeline for publishing the site both as text and over distributed protocols. The Techrights IPFS node over here (at home) now exceeds 50GB in traffic, just 3 weeks down the line (after introducing people to it, i.e. ‘going public’). Not bad considering how young it is. It’s always active, offering full site access. There’s no single point of failure, no HTTP, no WWW, no HTML. It’s very text-centric and thus compact, portable and so on. No bloat associated with exporting/importing images, fonts, JavaScript and so on.

Earlier today we published many pages of internal EPO material; we need to get those sorts of things backed up, as EPO management can be litigious and threatening (they tried it on us several times). Techrights is home to many other important bits of material; months ago we published old debian-private archives (1990s only); those were accessed nearly 30,000 times in this past week alone. Giving Debian better transparency 2.5 decades later can’t be a bad thing. In fact, nobody contacted us regarding removal or redaction. Those mailing lists are pretty harmless and barely even embarrassing. Due to their age, they don’t present/pose a threat to anybody’s career.

“In other words, the more people participate in this network, the more robust and censorship-resistant it’ll become. This, in turn, can attract more high-profile whistleblowers with high-profile leaks.”Now, on to the ‘beef’ of this post…

This past autumn we spent time coding and testing a bunch of stuff (at the back end for the most part). First we made everything in the site accessible in/as text. Plain text. Nice and simple. Then, we explored a number of distribution systems. At the end we went ahead of IPFS, seeing that it is decentralised and Web-agnostic (its protocols aren’t connected to the Web, unless a gateway is set up). We already have a number of devices pinning and serving the site’s pages (or objects) upon request by CID.

Share large logoHow can readers help? They can become nodes. The material itself isn’t sensitive (everything in it is public anyway), but it’s precarious in the sense that takedown requests can be attempted against our main servers; we want to make it very clear upfront that it’s an exercise in futility because many copies of the articles are already ‘out there’, being distributed by peer, not from a single point (of potential failure).

In other words, the more people participate in this network, the more robust and censorship-resistant it’ll become. This, in turn, can attract more high-profile whistleblowers with high-profile leaks.

An informal manual was typed up by one of us as a sort of primer for those wishing to set up their own node. It ought not be hard to achieve (by just following the series of steps). Those instructions were written for a Raspberry Pi with Debian, but the hardware and the distro ought not matter much because we use the binaries rather than repos.

“I’m going to outline a pragmatic setup that you can use to get going with IPFS on any host,” said the manual’s author. Here’s the recipe:


SUMMARY

Make user + group for IPFS. All further steps are to be performed under IPFS user’s “~/” (home directory).

Download and extract latest Go binaries available from official tarball.

Symbolic link Go binary executables, from tarball, into “~/bin“.

Download and extract latest IPFS binary available from official tarball.

Symbolic link IPFS binary executable, from tarball, into “~/bin“.

BEGIN IPFS BASIC USE GUIDE SUMMARY

What’s covered:

IPFS conceptual overview.

Help from the `ipfs` command itself.

Initialise ipfs for your IPFS user.

List pinned IPFS objects.

Add/pin IPFS objects.

Remove pinned IPFS objects.

Run garbage collector for IPFS objects.

Check IPFS stats.

Check a file’s CID without adding/pinning to IPFS.

SETUP DETAILS

Make a dedicated user and group for IPFS on your machine. This will keep things manageable, down the line. Everything should be done under the IPFS user’s home directory “~/“.

Grab the latest Go binary you can find. Set it up in the home directory of your IPFS user. You can find the latest Go binary here. There are builds for a wide array of operating systems and CPU architectures.

Extract the tarball; look in “/bin” in the tarball for the binary executables. Right now there are two binaries, “go” and “gofmt“.

Create a symbolic link in “~/bin” for the binary executables you find in the Go tarball. You should be able to run `$ source ~/.profile` to make sure “~/bin” is in PATH, for the IPFS user.

This completes the Go setup.

Next, we tackle the IPFS setup.

Download the latest binary you can get for go-ipfs from here. This binary should be packed in a tarball.

Extract the tarball.

go-ipfs” is the name of the binary executable. Create a symbolic link for this binary in “~/bin“. You don’t need to use the setup script provided. It’ll just mess with things that don’t need messing.

IPFS BASIC USE GUIDE

Think of IPFS as a filesystem that lives on the Internet. Each file is an IPFS object denoted by a hash called the CID (content identifier).

`ipfs --help` includes help for commands and subcommands. Always double-check with this to make sure that anything you read on the Internet about IPFS matches up to the binary you actually are using on your system.

Before you do anything with ipfs, make sure to run `ipfs init`. This will populate “~/.ipfs” for you. That’s all you need to do for initial setup.

You can “deploy” ipfs locally with `ipfs daemon`; this process will be running in the foreground and print to the terminal (probably stdout). You can just put it in the background and redirect stdout and stderr to a log file to manually monitor what it’s doing. Killing this daemon will mean your IPFS node is no longer online. Feel free to control this daemon in whichever method you choose. This guide is kept as abstract as possible to enable deployment in maximum number of environments.

Files from your host can be added to IPFS as something called IPFS objects. The ipfs command for this is, unsurprisingly, called `ipfs add`. `ipfs add path` will add the specified file or directory to IPFS.

By default, `ipfs add` “pins” objects in your local IPFS datastore. What is a “pinned object”? A pinned object is an IPFS object in your local IPFS datastore that doesn’t get garbage-collected.

`ipfs pin ls` is a good way to view your pins. `ipfs pin rm` can be used to remove pins; if you try to remove the wrong type of pin, IPFS will get mad and yell at you because it’s probably a recursive pin (you’ll see what kind of pin an IPFS object is in the output for `ipfs pin ls`); a top-level pin will probably be of type “recursive” (so you’ll need to remove that to get rid of all the pinned objects that are associated).

You can check bandwidth status with `ipfs stats bw`. Good way to keep track of your precious bandwidth. ipfs comes with some default pins. You might want to get rid of those with `ipfs pin rm` to save on bandwidth. Those pins are documentation, however, so it’s up to you if you want to keep them around or not.

Just removing a pin is not enough to ensure pin(s) are no longer is eating up your local IPFS datastore space. `ipfs repo gc` will run the garbage collector to get rid of all the unpinned IPFS objects.

The default IPFS configuration is pretty sane. You can find the default config file for IPFS in “~/.ipfs/config“; the config file is formatted in JSON. See `ipfs config --help` for further details.

Remember: always check `ipfs <command> <subcommand> --help` for definitive guidance on your binary’s implementation. `ipfs --help` should be skimmed, in full, before you do anything with IPFS; you will likely find commands to help you with what you want to do. Don’t forget that `ipfs <command> --help` and `ipfs <command> <subcommand> --help` are always available for you as well.

Consuming IPFS objects is straightforward. Just look at `ipfs get --help` and `ipfs cat --help` for instructions.

To check what a file’s CID is, without adding/pinning to ipfs, just run `ipfs add -n /path/to/file`; see `ipfs add --help` for details. This is going to be a very important operation if you want to do any automation. It helps knowing what a file’s CID will be before doing any operations live IPFS operations on it.

You should now be able to independently navigate the Internet for further information on IPFS. Official, online, documentation available here. Remember to always cross-check `ipfs version` and `ipfs <command> <subcommand> --help` to ensure what you read on the Internet is applicable to the IPFS binary executable on your system.

Image attributions: Font Awesome (Creative Commons Attribution 4.0 International) and IPFS project (Creative Commons Attribution-Share Alike 3.0 Unported)

Share in other sites/networks: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Reddit
  • email

If you liked this post, consider subscribing to the RSS feed or join us now at the IRC channels.

Pages that cross-reference this one

What Else is New


  1. InteLeaks – Part XXX: Harbor Research's Pseudo-scientific 'Research' for Intel, Bizarrely Suggesting a Microsoft Partnership for a Domain Largely Controlled or Dominated by Linux

    The full document that Intel paid for and in turn used to justify cracking down on Free software (obliterating Free software-based workflows inside Intel), instead outsourcing all sorts of things to proprietary software traps of Microsoft



  2. Chromium and Chrome Are Not Free Software But an Example of Microsoft-Fashioned Openwashing Tactics

    It's time to reject Google's Web monopoly (shared with other companies but still an oligopoly); removing its Web browser would be a good start



  3. Links 23/1/2021: Chromium Pains and New Debian Maintainers

    Links for the day



  4. InteLeaks – Part XXIX: Harbor Research Did Not Produce a Study But an Elaborate Hoax for Intel, Suggesting Microsoft Partnership and Outsourcing Based on Zero Evidence and No Solid Rationale

    The pseudo-scientific ‘report’ from Harbor Research is more of the same nonsense we’ve grown accustomed to; unethical if not rogue firms are being paid to lie — or to perpetuate falsehoods which someone stands to gain from



  5. Video: The State of Communities Surrounding GNU/Linux Distributions

    A discussion about the state of volunteer efforts going into the development, maintenance (in the 'maintainership' sense) and support/advocacy of GNU/Linux distros



  6. IRC Proceedings: Friday, January 22, 2021

    IRC logs for Friday, January 22, 2021



  7. InteLeaks – Part XXVIII: Intel Served Report From Microsoft Boosters, Who Provide No Actual Evidence and No Science to Back Their Supposed 'Findings'

    Findings and recommendations from Harbor 'Research' aren't based on any scientific methods, just perceived loyalty, branding, and a bunch of unsourced quotes (from unnamed people with ridiculous job titles like a soup of buzzwords)



  8. Erosion of Communities, Ascent of Corporate-Industrial Fake Communities

    Despite the attempts to manipulate/trick developers (and sometimes users) into becoming unpaid workforce of for-profit companies, there's an exodus back to real communities, which aren't subjected to the fury of wealthy shareholders who utterly dislike or simply don't care for software freedom



  9. The Corporate 'Left' and the Open Source Pseudo 'Movement'

    President Biden may not be as bad as his predecessor, but that hardly means very much; software freedom is still threatened, along with many other things



  10. Links 22/1/2021: pfSense Plus, Endless OS Foundation, and Many Laptops With GNU/Linux

    Links for the day



  11. The Linux Foundation is Trying to Obscure Racism Using Microsoft-Inspired Tactics (Vouchers Disguised as Actual Money)

    The Linux Foundation and its PR stunts don’t help combat racism; one might argue that the Foundation is leveraging racism, which prevails in the US, to paint itself as benevolent and caring (offering immaterial things and self-serving press releases)



  12. InteLeaks – Part XXVII: 'Pulling a Nokia' on Intel (Outsourcing to Microsoft)

    The recommendation of an Intel marriage with Microsoft (even in units that deal mostly with Linux) is an insulting slap across the face of developers employed there; we take a look at recommendations made to IoTG (Intel) by a firm with Microsoft orientation



  13. IRC Proceedings: Thursday, January 21, 2021

    IRC logs for Thursday, January 21, 2021



  14. InteLeaks – Part XXVI: Harbor Research is Horrible 'Research', Lacking Actual Technical Background

    Having looked at the members of staff of Harbor Research (individually), it seems clearer now why they have an affinity for Microsoft and why they're directing Intel to liaise with Microsoft and become a prisoner of Microsoft (even in areas where Microsoft is increasingly irrelevant)



  15. Links 21/1/2021: Raspberry Pi Pico, Ubuntu 21.04 Picks GNOME 3.38, KDE Plasma 5.21 Beta

    Links for the day



  16. How a Newly Inaugurated President Biden Can Advance Software Freedom (If He Actually Wishes to Do So)

    Techrights has 'Four Suggestions' to President Biden, the 46th 'front end' of American plutocracy



  17. InteLeaks – Part XXV: Intel's Brain Drain Leads to Unusual Measures

    As the company once known as 'chipzilla' loses its relevance and dominance in the market it's reaching out to retired people, trying to get them back onboard



  18. Hey Hi (AI) is Just a Trojan Horse for Illegal Software Patents, According to EPO Management and Litigation Firms It's in Bed With

    The longtime pushers or the lobby of patent profiteers just carry on pushing for software patents, nowadays latching onto the inane and unwarranted media hype around Hey Hi (AI) — a hype wave that was co-opted by EPO management to grant unlawful patents



  19. The Central Staff Representatives (CSC) of the EPO Are Petitioning to End the Assault on EPO Staff

    The EPO, just one month after the staff went on strike, is about to receive a compelling petition to stop the assault on EPO staff



  20. InteLeaks – Part XXIV: Love for Microsoft, Not for Free Software or Whatever Replaces Microsoft

    Intel is basing its big decisions on buzzwords and firms that master buzzwords; it's sad that instead of listening to Intel's own (in-house) engineers it's relying on a bunch of clowns who push 'Clown Computing' and 'apps' and 'UX'...



  21. IRC Proceedings: Wednesday, January 20, 2021

    IRC logs for Wednesday, January 20, 2021



  22. Links 21/1/2021: Google Tightens the Screws on Chromium, VideoLAN VLC 3.0.12

    Links for the day



  23. IBM Panics and Resorts to 'Customer Retention' Tactics With Red Hat Enterprise Linux (RHEL)

    IBM 'frees' RHEL but with limitations that can restrict growth of small companies (or subject them to financial barriers, originally unforeseen)



  24. Recent Techrights Articles About President Joe Biden

    Instead of writing yet more stuff about the latest US president, let's look back at what we wrote in recent weeks/months



  25. Links 20/1/2021: LibreOffice 7.1 RC2 and the RHEL Contingency

    Links for the day



  26. InteLeaks – Part XXIII: Intel Paying for Bogus 'Research' 'Insights' Which Merely Seek to Justify Outsourcing to Microsoft and Imposing Microsoft's Proprietary Software on Free Software Developers

    Intel's preference for Microsoft monopoly (an imposed/top-down decision) was seemingly certified by so-called 'consultants' and 'analysts' from the outside rather than the inside, basically manufacturing a false perception of consent after managers had already made up their minds



  27. Suppressed Facts of the Free Software Movement and Its Community of Volunteers – Part V: How FSF Secrecy Ended Up Insulting People, Alienating Trans Developers

    Having just uploaded this introductory video, we delve into the backstory or the real reason the FSF sank into somewhat of a crisis with the trans community almost half a decade ago



  28. InteLeaks – Part XXII: Bubbles and Buzzwords, No Substance at Intel's Internet of Things (IoT) Group (IOTG)

    The video above is continuation of the previous part about a document full of superficial buzzwords (not technical jargon anywhere), in effect recommending to managers that they blindly follow trends and cargo cults (such as Clown Computing) and not what’s most suitable for technical excellence



  29. IRC Proceedings: Tuesday, January 19, 2021

    IRC logs for Tuesday, January 19, 2021



  30. Links 20/1/2021: WireGuard for pfSense and New US President

    Links for the day


RSS 64x64RSS Feed: subscribe to the RSS feed for regular updates

Home iconSite Wiki: You can improve this site by helping the extension of the site's content

Home iconSite Home: Background about the site and some key features in the front page

Chat iconIRC Channel: Come and chat with us in real time

Recent Posts