08.09.19

Techrights Blog Archive and Word Analysis

Posted in Site News at 2:21 am by Dr. Roy Schestowitz

Summary: Summing up almost 13 years of Techrights and making archival simpler just in case something bad happens some time in the future

IN LIGHT of Linux Journal's shutdown, which was sudden and announced in the middle of the summer holidays (may have been strategic timing), we’ve already published a list of Techrights blog URLs and our full Wiki as XML files (or compressed archive of them all). That’s not because we expect anything bad to happen; it’s just better to be prepared in advance rather than when it’s too late. I’ve poured my entire adult life into this site and the analysis/research it takes to write/run it. It’s good to have backups, including elsewhere. Mirroring does no harm, it improves overall resilience when things are decentralised. During the archiving process figosdev wrote some code that we have not published until today. It’s used to generate a keyword index with merges for similar ones. So for example, for the latter it comes up with (number of occurrences on the left):

14191 chrome 9611:chrome. 233:chromebook 1754:chromebook. 121:chromebooks 1513:chromebooks. 156:chromebox 74:chromecast 396:chromeos 333:
14195 e.u. 109:eu 13203:eu-us 107:eu-wide 144:eu. 632:
14271 process 11848:process. 2423:
14278 china 13069:china-based 95:china. 1114:
14371 receive 3588:received 7239:received. 141:receives 1405:receiving 1998:
14411 firefox 14411:
14556 download 9003:download. 1015:downloadable 220:downloaded 1417:downloader 101:downloaders 97:downloading 1100:downloads 1454:downloads. 149:
14739 gates 14158:gates-backed 114:gates-funded 141:gates. 326:
14850 digital 14577:digital. 86:digitally 187:
14894 employed 938:employee 2773:employee. 154:employees 6620:employees. 704:employer 989:employer. 168:employers 644:employing 284:employment 1158:employment. 93:employs 369:
15026 package 6369:package. 512:packaged 533:packagers 85:packages 5762:packages. 767:packaging 998:
15170 menace 148:menacing 100:menos 418:mental 1124:mentality 273:mentally 216:mention 4183:mention. 103:mentioned 6743:mentioned. 155:mentioning 720:mentions 987:
15196 games 14158:games. 1038:
15231 event 7653:event. 954:events 6020:events. 604:
15350 corporate 8067:corporates 77:corporation 2759:corporation. 240:corporations 3790:corporations. 417:
15419 battistelli 14513:battistelli. 906:
15560 browser 9915:browser. 1333:browsers 4024:browsers. 288:
15709 privacy 13613:privacy-focused 124:privacy. 992:privacy/surveillance 980:
15749 press 14852:press. 897:
15759 require 3825:required 4340:required. 380:requirement 1182:requirement. 139:requirements 2226:requirements. 480:requires 3187:
15932 intel 15517:intel-based 101:intel. 232:intel/amd 82:
16050 police 15513:police. 537:
16695 mobile 16432:mobile. 263:
16877 troll 4535:troll. 434:trolles 205:trolling 1127:trolling. 111:trolls 9276:trolls. 1189:
16880 enterprise 13684:enterprise-class 132:enterprise-grade 221:enterprise-ready 131:enterprise. 424:enterprises 2072:enterprises. 216:
16985 hardware 15321:hardware. 1664:
17492 policies 4017:policies. 638:policy 11524:policy. 1313:
17832 reported 9524:reported. 848:reportedly 3159:reporter 1927:reporter. 94:reporters 2141:reporters. 139:
17865 user 16688:user. 826:userbase 82:username 174:usernames 95:
18106 federal 18106:
18157 money 16399:money. 1758:
18170 result 6842:result. 452:resulted 1431:resulting 1346:results 7119:results. 980:
18209 management 17099:management. 1110:
18362 research 11250:research. 727:researched 153:researcher 1346:researchers 4435:researchers. 196:researching 255:
18746 review 12483:review. 723:reviewed 1304:reviewer 149:reviewers 230:reviewing 721:reviews 2931:reviews. 205:
18993 censor 1588:censored 952:censored. 74:censoring 859:censors 823:censorship 12500:censorship. 701:censorship/free 972:censorship/privacy/civil 404:censorship/web 120:
19233 debconf 276:debian 17855:debian-based 626:debian. 476:
19233 name 10460:name. 764:named 4402:names 3607:
19371 facebook 18648:facebook. 723:
19681 question 9688:question. 879:questioned 808:questioning 729:questions 6938:questions. 639:
19804 order 14028:order. 692:ordered 2399:ordering 431:orders 2047:orders. 207:
19861 organisation 3331:organisation. 393:organisational 97:organisations 1789:organisations. 194:organization 7031:organization. 717:organizational 290:organizations 5484:organizations. 535:
20396 national 20321:national-security 75:
20636 member 7943:member. 320:members 11588:members. 785:
21621 copyright 16583:copyright. 470:copyrighted 710:copyrights 3676:copyrights. 182:
21704 official 11777:official. 211:officials 9084:officials. 632:
21809 rights 18893:rights. 1936:rights/policing 980:
21944 cloud 20228:cloud-based 661:cloud. 1055:
21978 nsa 21533:nsa. 445:
22096 industrial 2497:industries 1720:industries. 357:industry 15436:industry. 2086:
22146 kde 21049:kde. 488:kde3 72:kde4 537:
22170 fedora 21663:fedora. 507:
22411 standard 8863:standard. 889:standardisation 232:standardised 77:standardization 422:standardize 182:standardized 329:standardizing 80:standards 9113:standards-based 123:standards. 1265:standards/consortia 836:
22416 market 16524:market. 2932:markets 2361:markets. 599:
22428 protect 6965:protected 2043:protected. 142:protecting 2235:protection 8152:protection. 633:protections 1715:protections. 307:protective 236:
22526 political 13646:politician 758:politicians 3860:politicians. 247:politico 350:politics 3118:politics. 547:
22664 product 8764:product. 1060:products 10698:products. 2142:
22994 publish 2200:published 13513:published. 392:publisher 1348:publisher. 70:publishers 1654:publishers. 127:publishes 825:publishing 2757:publishing. 108:
23532 next 23174:next. 358:
23594 permalink 23594:
23726 foundation 22066:foundation. 1660:
23765 opensource 268:open-source 22369:open-source. 110:open-sourced 387:open-sources 402:open-sourcing 229:
24419 gnome 23574:gnome-based 84:gnome-shell 123:gnome. 497:gnome3 141:
24981 video 18925:video. 596:video> 271:videolan 113:videos 4794:videos. 282:
25022 icons 24892:icons. 130:
25227 plan 8738:plan. 578:planned 3084:planned. 168:planning 3385:planning. 77:plans 8758:plans. 439:
25671 attack 10039:attack. 819:attacked 1573:attacker 739:attackers 1018:attacking 1986:attacks 8438:attacks. 1059:
26054 network 14441:network. 1261:networked 234:networking 4706:networking. 186:networks 4378:networks. 848:
26394 countries 9065:countries. 1438:country 13370:country. 2521:
26827 information 24642:information. 2185:
27074 phone 16072:phone. 891:phones 9315:phones. 796:
28926 http 28821:http/2 105:
29542 posted 28779:posted. 94:poster 429:posters 240:
29681 server-side 196:server 19956:server. 1406:servers 6949:servers. 1174:
29914 office 27490:office. 2424:
30007 platform 19050:platform. 3365:platformer 484:platforms 5649:platforms. 1459:
30132 development 26183:development. 2005:developments 1744:developments. 200:
30137 code 25179:code. 3191:coder 167:coders 349:coding 1168:coding. 83:
30291 media 28467:media. 1824:
30445 feature 9058:feature. 481:features 18736:features. 2170:
30791 kernel 27263:kernel. 2250:kernel-based 284:kernels 870:kernels. 124:
31184 communities 3334:communities. 737:community 23348:community-based 105:community-driven 193:community. 3467:
31828 u.s 201:u.s. 28527:u.s.-backed 113:u.s.-based 132:u.s.-led 93:us-backed 91:us-based 401:us-led 135:usa 1960:usa. 175:
32042 used 31372:used. 670:
32990 program 13901:program. 2001:programmed 179:programmer 803:programmers 1432:programmers. 143:programmes 381:programming 5917:programming. 237:programming/development 818:programs 6212:programs. 966:
33607 reader 4579:reader. 278:readers 28308:readers. 347:readership 95:
34045 internet 30644:internet. 2458:internet/net 943:
34523 america 11784:america. 1032:american 14414:american. 80:americans 6414:americans. 589:americas 210:
34826 page 6591:page. 1001:pages 3250:pages. 23984:
34864 technological 1251:technologically 152:technologies 7327:technologies. 1290:technology 22387:technology. 2457:
35836 presidency 1256:presidency. 275:president 27787:president. 1084:presidential 4834:presidents 600:
37154 distribution 12395:distribution. 1967:distributions 12171:distributions. 1095:distro 5812:distro. 652:distros 2743:distros. 319:
37446 epo 35039:epo. 2075:epo.org 332:
37687 public 35751:public. 1936:
37723 social 37600:social-media 123:
37846 gnu 6426:gnu. 93:gnu/hurd 81:gnu/kfreebsd 85:gnu/linux 29520:gnu/linux-based 91:gnu/linux. 1550:
38367 novell 36653:novell. 1714:
39579 developer 13599:developer. 373:developers 23843:developers. 1764:
40375 case 27904:case. 2515:caselaw 108:cases 8853:cases. 995:
41238 version 33068:version. 1440:versioned 146:versioning 164:versions 5822:versions. 598:
41601 users 37295:users. 4306:
42802 europe 16585:europe. 1902:european 22948:europeans 444:euros 923:
42826 court 34888:court. 1976:courts 5330:courts. 632:
43149 photon 97:photos 2662:photos. 191:photoshop 483:php 2278:php-fpm 105:php5 164:phpmyadmin 164:phrase 911:phrases 272:physical 2586:physically 388:physician 130:physicians 160:physicist 95:physics 466:physx 70:pi 7518:pi. 361:piana 111:pic 72:pichai 173:pick 1939:picked 1183:picket 100:picking 722:picks 900:pico 71:pico-itx 156:pics 99:picture 2355:picture. 321:pictured 311:pictures 1450:pictures. 80:pidgin 230:pie 1637:pie. 72:piece 5026:piece. 171:pieces 2059:pieces. 132:pierre 180:pierre-yves 94:pieter 113:pig 220:pigs 193:pike 97:pile 578:piled 78:piles 129:pilger 118:piling 112:pill 149:pillar 135:pillars 217:pilot 988:pilots 320:pim 201:pimping 70:pin 272:pinch 120:pine 176:pine64 118:
43445 reporting 4831:reporting. 240:report 26127:report. 1583:reports 9493:reports. 1171:
43555 update 20736:update. 1135:updated 8817:updated. 210:updatedx2 85:updater 93:updates 10442:updates. 999:updating 1038:
43583 desktop 36980:desktop. 2291:desktop/gtk 1351:desktops 2468:desktops. 493:
49940 win32 90:windows 46655:windows-based 204:windows-only 192:windows. 2799:
50033 project 32819:project. 3734:projects 11447:projects. 2033:
51865 government 43803:government. 2416:governmental 467:governments 4822:governments. 357:
54220 web 52026:web-based 864:web-browser 150:web. 1180:
54986 goog 218:google 53003:google+ 410:google. 1355:
55737 secure 7203:secure. 417:security 46115:security-focused 77:security. 1925:
60088 support 49775:support! 74:support. 3530:supported 6428:supported. 281:
64412 android 60980:android-based 684:android-powered 403:android-x86 196:android. 1650:android/linux 499:
66673 patenting 1475:patenting. 120:patents 58590:patents. 6408:patents] 80:
68988 people 65485:people. 3115:peoples 388:
71735 ubucon 83:ubuntu 68622:ubuntu! 86:ubuntu-based 643:ubuntu-powered 183:ubuntu. 1791:ubuntu/linux 327:
82416 patent 79808:patent. 1178:patented 1245:patented. 185:
111622 release 54907:release! 153:release. 5083:released 36637:released! 844:released. 1621:releases 11390:releases. 987:
117419 who 86708:why 30100:why. 611:
123764 open 123225:open. 539:
128149 software 111593:software-as-a-service 94:software-based 127:software-defined 636:software-like 535:software-related 106:software. 8488:software/open 6570:
165377 msft 784:microsoft 157420:microsoft-connected 384:microsoft-friendly 239:microsoft-funded 276:microsoft-influenced 73:microsoft-novell 147:microsoft-sponsored 129:microsoft-taxed 102:microsoft. 5451:microsoft/novell 227:microsoft] 145:
199787 linux 189841:linux! 329:linux- 121:linux. 8892:linux-related 104:linux/android 266:linux/unix 150:linux] 84:

The underlying code, trc.fig (in Fig):

#### license: creative commons cc0 1.0 (public domain) 
#### http://creativecommons.org/publicdomain/zero/1.0/ 
# load log.txt (db of entries) and search from trc.txt (output from coll.fig)

python
    reload(sys)
    sys.setdefaultencoding('utf8')
    fig

nl 10 chr
now "" arr
nowlen arr

function getlink p
    eachr p ltrim
    pg split eachr nl mid 1 1
    pr split pg "ignorethis " join pr "" return pr
    fig

function rplace p c t
    now split p c join now t return now
    fig

function ctext t
    quot 34 chr
    nl 10 chr
    tab 9 chr
    now t lcase rplace now quot nl swap now t
    now t rplace now " " nl swap now t
    now t rplace now "&" nl swap now t
    now t rplace now ";" nl swap now t

    now t rplace now ")" nl swap now t
    now t rplace now "(" nl swap now t
    now t rplace now "," nl swap now t
    now t rplace now "'" nl swap now t
    now t rplace now ":" nl swap now t
    now t rplace now "?" nl swap now t
python
    t = t.replace(unichr(8220), chr(10))
    t = t.replace(unichr(8221), chr(10))
    fig
    now t rplace now tab nl swap now t return t
    fig

function nohtml p
    buf ""
    intag 0 
    forin each p
        ifequal each "<"
            intag 1
            fig
        ifequal intag 0
            now buf plus each swap now buf
            fig
        ifequal each ">" 
            intag 0
            fig
        next
    now return buf
    fig

function lookfor findthese db dbcopy origfind
    nl 10 chr
    dblen db len
    buf 1
    for eachlen 1 dblen 1
        each arrget db eachlen
        copyeach arrget dbcopy eachlen

        forin eachofthese findthese
            iftrue eachofthese
                #now eachofthese plus "-" print
                now each split now nl
                found 0
                found instr now eachofthese
                iftrue found

                    iftrue buf
                        now origfind print # print current list of searchwords
                        buf 0
                        fig

                    now "*[[" prints
                    now copyeach split now "\n" mid 6 2 
                    eurl getlink copyeach prints "|" prints
                    etitle now mid 2 1 ltrim rtrim
                    edate now mid 1 1 ltrim rtrim plus " " plus etitle plus "]]" print
                    iftrue 0
                        now eachofthese print # print found keyword
                        fig
                    break
                    fig

                fig
            next

        next
    fig



db "" arr
dbcopy "" arr
dblen 0
python 
    db = open("log.txt").read().split("If you liked this post, consider")
    dbcopy = db[:]
    dblen = len(db)
    fig

now "processing entries... " prints
prevcount 0
for eachdb dblen 1 -1
    dbcount eachdb divby dblen times 1000 int divby 10 minus 100 times -1

    ifmore dbcount prevcount
        now dbcount str plus "% " prints
        prevcount dbcount
        fig 
    p arrget db eachdb 
    now nohtml p ctext now
    db arrset eachdb now

    p arrget dbcopy eachdb 
    now nohtml p
    dbcopy arrset eachdb now

    next
now "" print

pg arropen "trc.txt" 
nowlen pg len

for eachloop nowlen 1 -1
    each pg mid eachloop 1   
    eaches split each " " 
    eacheslen eaches len minus 1
    now "" print
    reaches eaches right eacheslen join reaches " " split reaches ":" 
python
    reaches.remove('')
    fig

    iftrue reaches
        findthese arr mid 1 0 
        forin eachreaches reaches
            onereach split eachreaches " " mid 1 1 
            findthese plus onereach 
            next
        iftrue findthese
            now lookfor findthese db dbcopy each
            fig
        fig

   next

And coll.fig

#### license: creative commons cc0 1.0 (public domain) 
#### http://creativecommons.org/publicdomain/zero/1.0/ 
# combine trranks into trc.txt (with > trc.txt)
fl arropen "trranks"
buf "" arr mid 1 0
buftext ""
bufsum 0
forin each fl
    lentr each rtrim len
    iftrue lentr
        buf plus each
    else
        iftrue buf
            forin beach buf
                now buftext plus beach plus ":" swap now buftext
                teg split beach " " mid 1 1
                rank split beach " " mid 2 1 val
                now bufsum plus rank swap now bufsum
                next
            now bufsum prints " " prints
            now buftext print 
            buf "" arr mid 1 0
            buftext ""
            bufsum 0
            fig
    fig
    next

There’s more code we’ve yet to share. Just needs more tidying up. We want everything to be open data, Free/libre software, Open Access etc. It takes time. █

Permalink  Send this to a friend