Centre for Internet and Society

Querying Wikipedia Data

pavanaja — 2016-10-21T14:51:01Z

Recently I wrote a blog about the stub article length of Wikipedia articles. I mentioned the difference in actual number of characters and the number of bytes used to define stub articles between English and Indian language Wikipedias. One can open any language Wikipedia, type Special:ShortPages in the search box to get the list of articles which have less than 2048 bytes.

Originally published blog can be accessed here.

However, as already mentioned in that blog, the number of bytes for Indian languages to be considered as stub should be actually 2048*3 = 6144 bytes employing the same criteria. How to find the list of articles fulfilling this condition?

This brings us to the topic of querying Wikipedia data. Wikimedia Foundation Labs has put up a website wherein one can run SQL queries on Wikimedia data. The URL of the website is quarry.wmflabs.org. When we open the website, we get a textbox wherein one can type the SQL query which will run on Wikimedia data. In this example I will consider Wikipedia only. But the queries can be run on the data of other Wikimedia projects like Wikisource, Wikidata, Wiktionary, etc.

One has to login with his/her Wikimedia login. After logging the SQL query can be typed in the textbox and the Submit Query button has to be clicked. The result of execution of the query on Wikimedia data will be displayed. In this blog I will throw more light on Kannada Wikipedia. The database for Kannada Wikipedia is called knwiki_p. Complete list of databases can be obtained by running the SQL query “show databases”.

To get the list of tables in Kannada Wikipedia, the following SQL queries have to be executed:

use knwiki_p;
show tables;

To know the scheme of any table, run the query desc <tablename>;. For example, to know the details of the table by name, issue the query desc page;. The fields which are of importance in the current case is page_title and page_len. The following query will list all articles in Kannada Wikipedia which are having less than 6144 bytes.

use knwiki_p;
select page_title, page_len
from page where page_len < ‘6144’ and page_namespace = 0 and page_is_redirect = 0 order by page_len ;

Some other useful queries are listed below:

Query	What it does
Select Count(*) from page where page_namespace = 0 and page_is_redirect =0;	Number of articles without redirect
Select Count(*) from page where page_namespace = 0 and page_is_redirect =0 and page_len < 6144;	Number of articles which are having bytes less than 6144
select * from user where user_name Like “P%”;	List all users whose username starts with letter “P”
select user_id, user_name, user_editcount from user where user_editcount >3000 order by user_editcount desc;	List all users with editcount more than 3000
SELECT page_namespace, page_title, page_len FROM page WHERE page_len > 175000 AND page_title NOT LIKE “%/%” ORDER BY page_namespace ASC;	List of long articles (articles having bytes more than 175000)
SELECT rc_title as title, rc_comment as comments, count(*) as Edits FROM recentchanges WHERE rc_namespace = 0 GROUP BY 1 ORDER BY 3 DESC LIMIT 100;	Most edited 100 pages during past one month
SELECT log_title, COUNT() FROM logging WHERE log_type=”thanks” GROUP BY log_title ORDER BY COUNT() DESC LIMIT 100;	Who have been thanked most

Useful links -

For more details visit https://cis-india.org/a2k/blogs/querying-wikipedia-data

UTF-8, Indic and Stub Length Article in Wikipedia

pavanaja — 2016-10-20T02:26:29Z

One of the activities conducted as part of Wiki Conference India 2016 was the Punjab Editathon. It was about adding articles related to Punjab to Indian language Wikipedias and English Wikipedia. There was also an announcement made about some award for highest contribution.

See the original blog post at Dr. Pavanaja Blog

This lead to continued discussions in a closed chat group on how do we decide the winner. People thought it is very simple to announce the winner just based on highest number of bytes added. On first look, it looked very trivial and a simple case. I pointed out during the discussions about the encoding used in Wikipedia is UTF-8 and it uses different number of bytes for English and Indian languages. Before giving more details I would like to draw your attention to a simple experiment.

I typed Kannada letter ಅ (a) in my Sandbox in Kannada Wikipedia and saved it. Then I checked the RecentChanges page in Kannada Wikipedia. That showed that I have added 3 bytes to my Sandbox page. But I had added just one Kannada character. I did the same experiment in English Wikipedia. I just added one letter, the English letter “A” to my Sandbox in English Wikipedia and checked the number of bytes added. It showed just one byte.

What is going on? Here is the explanation. There are different ways Unicode text can be stored. UTF-8, UTF-16 and UTF-32 are the prominent ways. UTF-16 uses 2 bytes for all characters. UTF-32 uses 4 bytes. UTF-8 is a special kind of encoding. It uses series of single bytes to represent Unicode data. The first character, called Byte Order Mark (BOM) indicates what encoding is being used. Unicode website has more details on these. UTF-8 was mainly used for web as the networking devices used on the initial days of Unicode could handle only 8 bits (1 byte) of data. In other words, UTF-8 was used for backward compatibility with ASCII, the original 8-bit encoding used prior to the advent of Unicode. Even today the default encoding used by HTML is UTF-8.

Does these answer our original question? Not yet. I said UTF-8 uses series of single bytes. It uses 1 byte for English, 2 bytes for European languages and 3 bytes for Indian languages. That is the reason why we saw 3 bytes for one Kannada character.

This pops up another interesting question regarding the definition of a stub article in Wikipedia. As per Wikipedia, an article which has less than 2048 bytes is considered as a stub article. Go to any language Wikipedia’s search page and type Special:ShortPages to get the list of all articles which are having less than 2048 bytes. If we convert this into number of characters it turns out to be 2048 for English but about 682 for Indic. That means the length of a stub article will be different for English and Indian language Wikipedias. Should we have a different yardstick for the definition of a stub article for Indian language Wikipedias then? I think yes.

For more details visit https://cis-india.org/a2k/blogs/utf-8-indic-and-stub-length-article-in-wikipedia

Localisation experiences in Kannada and Tulu Wikipedias

pavanaja — 2016-10-06T01:51:02Z

It has become a fashion to talk about the digital divide in India. These talks are going on for more than 2 decades now. Nevertheless there is no doubt about the need of taking the benefits of Information Communication Technology (ICT) for common man.

Originally published in my blog on October 4, 2016

FUELling Indic computing

The statement “ICT for common man has to be in his language” is always relevant. Many state governments and the central government of India are spending lot of money towards E-Governance. Of late many private vendors also have realised the importance of localising their applications into Indian languages. Be it a web based or mobile app based solution providing it in Indian language will increase the user-base by huge magnitudes.

This brings us to the issue of standardisation of terminologies used in localising the ICT applications into Indic. How does one write frequently used terms in Indic? Is there any standard? For example, how to write “Exit” in Kannada? Should I use “ನಿರ್ಗಮನ” or “ಹೊರಗೆ”? Enter FUEL Project. This project aims at standardising the Frequently Used Entries in Localisation. The project aims at having a consistency among the terms used across all applications, apps and websites. FUEL Project is run by volunteers. Corpus of translations of the terminologies is developed by contributions of enthusiastic people in true opensource spirit.

FUEL project has been organising annual conferences referred as Globalisation Internationalisation Localisation and Translation, in short GILT Conference. This year’s GILT Conference was held at New Delhi, India, during Sep 24 & 25, 2016. This year’s conference had general users, media persons and writers who are nothing but the normal consumers of the output of FUEL usage apart from the techies. There was a track exclusively dedicated for Mozilla localisation. The detailed agenda and speakers list is available here.

Localisation experiences in Kannada and Tulu Wikipedias

I have been a Wikipedian for more than a decade. I have contributed to Kannada Wikipedia ever since it was started way back in the year 2004. Tulu Wikipedia went live recently which was in incubation from 2008. I have been contributing to Tulu Wikipedia as well. I presented a paper at FUEL GILT Conference titled “Localisation experiences in Kannada and Tulu Wikipedias”.

Kannada and Tulu belong to Dravidian family of languages. Geographically and politically they both belong to Karnataka. Both languages share a lot of things and there are many common words used in both languages. At the same time Kannada and Tulu have their own distinct identities. One striking feature is that Tulu has less Sanskrit words compared to Kannada. Tulu had its own script and grammar but over the years the usage of Tulu script has been discontinued. Nowadays most Tulu people use Kannada script. This has led to the usage of many Kannada words in place of original Tulu words.

Kannada has an established history of science and technology writings as well as localised software applications. Naturally a well-developed glossary is available. It was somewhat easy to localise the strings used in Mediawiki, which is the platform used to host Wikipedias, into Kannada, which very fall under FUEL category. In Kannada, we have been using these formulas while creating technical glossary –

Create altogether new term
Translate the English term
Transliterate the English term, i.e., write the English term in Kannada script

Depending on the situation, one of the 3 paths is followed. The same philosophy is extended for localising the strings used in Mediawiki. In many situations, we have either a part translation available or a similar example available in Kannada which made the localisation of Mediawiki not so difficult.

Tulu Wikipedia was in incubation since 2008. It became live on Aug 05, 2016. One of the main requirement to make it live was to have major strings translated into Tulu. Most of the commonly used strings have been translated now. But the case of Tulu is not so simple which has no history of science and technology writing. It was quite difficult to localise most of the strings. At many places Kannada strings have been used. Many more strings are yet to be localised. In case of Tulu, we have been using these formulas for Mediwaiki translations –

Create altogether new term in Tulu
Use more of Kannada words in the localised string
Transliterate the English term, i.e., write the English term in Kannada script. Since Tulu is using Kannada script, the strings thus used will be the same for Kannada and Tulu

The questions and answers just after the presentation had some very interesting questions from other language Wikipedians as well as people from other walks of life who are into localisation. I could benefit by interacting with others and by gaining insights into some unexplored areas by me. Interactions with other participants which also included some Wikipedians continued after the presentation. Overall it was quite useful spending of two days. Looking forward for the next event like this.

Acknowledgements

Kannada and Tulu Wikipedians
Dr Vishwnatha Badikana – Tulu translations
Bharathesha Alasandemajalu -Tulu translations
Wikimedia Foundation for sponsoring the travel

Links:

For more details visit https://cis-india.org/a2k/blogs/pavanaja.com-english-fuel-gilt-kannada-tulu

WikiConference India 2016 Submission: Gender Gap in Indian Language Wikipedias – Projects, Outreach and Outcome

pavanaja — 2016-08-17T02:43:54Z

India is not much different from the global statistics when it comes to gender gap issues in Wikipedia. Indian language Wikipedias are known to have similar ratio of male to female editors.

The submission was included as part of a panel related to the gender gap at the WikiConference India 2016.

Many activities are being made to bridge the gender gap in Indian language Wikipedias. These are aimed both at improving the ratio of female to male editors and increasing women related content in Indian language Wikipedias. Some of the notable projects are Art+ Feminism editathon, Lilavati’s daughters editathon, International Women’s day or month editathon, editathon to add articles about woman writers of coastal Karnataka. Sizeable number of students who are part of the Wikipedia in Education Program which are being run at many educational institutes of India are female. Some of these students continue editing and contribute to Wikipedia. Some female students worked as interns and have contributed by improving the quality of articles written by their peers. Outreach is one of the regular activities of taking Wikipedia to people. Outcome of the projects and the data on retention of the participants of outreach activities as regular contributing Wikipedians, survival of the newly created editors for short and long periods are documented and will be presented at the conference.

For more details visit https://cis-india.org/a2k/blogs/wikiconference-india-2016-submission-gender-gap-in-indian-language-wikipedias-2013-projects-outreach-and-outcome

Kannada Wikipedia edit-a-thon in St. Agnes College, Mangaluru to bridge gender gap in Wikipedia

pavanaja — 2016-01-12T11:22:25Z

As part of a series of workshops to mark the 13th anniversary, Kannada Wikipedia, a Kannada Wikipedia edit-a-thon was organised during this 10th-12th in St. Agnes College, Mangaluru to address the issue of gender gap in Wikipedia.

Totally 24 female students Wikimedians participated in this edit-a-thon. Three female Wikipedia editors from the institution also joined as facilitators. Vishwanatha Badikana, Kannada professor of the institution who is also a Kannada and Tulu Wikipedian inaugurated the event as chief guest. The first day was spent in demonstrating about the Wiki markup. During the following two days the participants created articles about notable writers and other notable women of coastal Karnataka. 25 new articles were created by the participants on the second day. By the end of the last day, 25 more articles were created in sandboxes on the last day by lunch time. They will soon work on the articles and improve before moving them to the article namespace so they become live articles. From the three day long edit-a-thon about 75 new articles are expected to come on Kannada Wikipedia. More details about the edit-a-thon could be read on Kannada Wikipedia event page.

For more details visit https://cis-india.org/openness/kannada-wikipedia-edit-a-thon-in-st-agnes-college-mangaluru-to-bridge-gender-gap-in-wikipedia

A Kannada Wikipedia brochure for newbies

pavanaja — 2015-10-19T10:46:38Z

This brochure has been created keeping readers and new Wikipedia editors in mind. This would come handy to learn editing Kannada Wikipedia and creating/editing articles in Kannada.

What does it contain?

How Wikipedia is world’s largest free and open encyclopaedia?
Did you know that "anyone can edit Wikipedia"?
How Wikipedia helps in preserving the knowledge and taking it to the next generation. For our languages to survive and grow, it is very important to bring world knowledge in our language. Wikipedia helps in expanding that knowledge repository as anyone can contribute to Wikipedia.
There is no financial gain by writing in Wikipedia. It is very easy to start editing Kannada Wikipedia. Open kn.wikipedia.org, check the help file, tutorial file & videos, start editing in your sandbox (learn more about the Wikipedia sandbox).
No poetry, blog, story, criticism, writing about self, advertisements of any sorts, etc are not allowed in Wikipedia

Download this brochure from here.

For more details visit https://cis-india.org/openness/blog-old/a-kannada-wikipedia-brochure-for-newbies

WikipediansSpeak with Dr. Kishore Kumar Rai Sheni: Tulu Wikipedia in the making

pavanaja — 2016-06-18T18:06:49Z

Dr. Kishore Kumar Rai Sheni, Principal, Shri Ramakrishna P.U. College, Mangaluru shares with Dr. U.B. Pavanaja of the Centre for Internet and Society's Access To Knowledge (CIS-A2K) programme his insights on bringing more contributors of different corners of the society to grow Tulu Wikipedia. Tulu Wikipedia is in Incubator at present and there are a bunch of contributors who are expanding the project to bring it live.

When I caught up with Dr. Kishore Kumar Rai Sheni during an ongoing Tulu Wikipedia edit-a-thon in his college, Shri Ramakrishna P.U. College, Mangaluru, he left me with his wonderful ideas of expanding the Tulu Wikipedia. Back in 2014, with a few sporadic contributors, the Tulu Wikipedia community probably did not exist. After several outreach programmes in and around Mangaluru, it has grown to about an eight-editor community. It could look really tiny today but has potential to grow to a bigger community provided the constant effort by these eight people continue. This edit-a-thon is one of the stepping stones to build this community. About 37 people who hardly know anything about Tulu Wikipedia are participating in this event where six of the experienced editors will support them. The project is all about creating a live encyclopedia on the internet. We have enough and many Tulu authors and teachers who otherwise contribute to the language and literature. The idea is to bring them aboard. In collaboration with three other institutions here; Ramakrishna PU College, Ramakrishna Degree College and Tulu Sahitya Academy, we're trying to bridge the gap of lack of Wikipedia in our language. So far, very little work has been done in documenting the rich Tulu culture in Tulu language. There is no encyclopaedia of any sort in Tulu at this moment. There is also a need to bring the entire world's knowledge to the Tulu speakers. I hope Tulu Wikipedia will be the future to take our language and culture to the world, and fill the gap of knowledge that exists today.

Tulu Wikipedia Incubator is available at https://incubator.wikimedia.org/wiki/Wp/tcy.

For more details visit https://cis-india.org/openness/blog-old/wikipediansspeak-with-dr-kishore-kumar-rai-sheni-tulu-wikipedia-in-academics

Journalism Students of the SDM College Ujire Enrich Karnataka’s Folklore And Folk Art in Kannada Wikipedia

pavanaja — 2015-09-15T09:09:12Z

As part of an ongoing partnership with the Shree Dharmasthala Manjunatheshwara (SDM) College Ujjire, with active support from a few Kannada Wikipedia editors, CIS-A2K began an outreach programme so that the journalism students could help many Kannada readers about Karnataka’s rich folklore and folk art.

Both first year and second year students of Master of Communication and Journalism (MCJ) of SDM College participated in this workshop. Out of 35 participants, 11 were female. Students had discussed already about enhancing Kannada Wikipedia articles on folklore and folk art forms of Karnataka. About 20 new user accounts were created and the students have started creating articles in their user sandboxes which they will later move as articles upon enhancement with vital information. Some of the students chose to find existing articles and add more information to them. Long time Kannada Wikimedian S N Vasanthkn. from Dharmasthala helped as resource person to help the new editors with Wikipedia editing. However, as first timers, many struggled with the encyclopedic way of writing and maintaining neutral point of view. These students will be mentored by Vasanth as he is visiting them every Monday.

More details from the event page.

For more details visit https://cis-india.org/openness/blog-old/journalism-students-of-the-shree-dharmasthala-manjunatheshwara-sdm-college-ujire-enrich-karnataka2019s-folklore-and-folk-art-in-kannada-wikipedia

Kannada Wikipedia workshop at Shree Dharmasthala Manjunatheshwara (SDM) College Ujire, Karnataka

pavanaja — 2016-06-18T18:12:41Z

A Kannada Wikipedia workshop is being planned to be organised in Shree Dharmasthala Manjunatheshwara (SDM) College Ujire on the 13th August 2015. This is going to be part of the several outreach programmes in the institution.

Topics to be discussed:

What, Why & How of Wikipedia
Basics of Wikipedia markup
Kannada Wikipedia
Editing Kannada Wikipedia
What can be added to Kannada Wikipedia and how to add articles

For more details visit https://cis-india.org/openness/events/kannada-wikipedia-workshop-at-shree-dharmasthala-manjunatheshwara-sdm-college-ujire-karnataka

Workshop on digital collaborations in Tamil-language, Tamil Virtual University, Chennai

pavanaja — 2016-06-18T18:13:56Z

A two-day workshop on digital collaborations in Tamil-language has been organised by Tamil Virtual University, Anna University Campus, Chennai. I have been invited to participate in this event on behalf of the CIS-A2K programme.

The discussions are going to be in four major topics:

Digitization:

Books and Digital Library
Manuscripts, Inscriptions, Documents, Paintings and Archival of documents

Research and Development:

Natural Language Process
Video / Speech synthesizer, Spell checker
Corpus and Lexicon

E-learning:

Developing appropriate and region specific syllabus
Mobile computing and applications development

Open source and Outreach:

Tamil computing organisations and Wikipedia
Policy of Open source software

For more details visit https://cis-india.org/openness/events/workshop-on-digital-collaborations-in-tamil-language-tamil-virtual-university-chennai

Wikipedia edit-a-thon in Mangalore to bring Tulu Wikipedia live

pavanaja — 2015-09-15T09:07:24Z

A Wikipedia edit-a-thon was organised in Mangalore, Karnataka this 14th to encourage more Tulu-language speakers to contribute to Tulu Wikipedia. Tulu Wikipedia is is currently in the Incubator but the enthusiastic editor community is putting their best effort to bring it live out of Incubator. This edit-a-thon is one of the many activities the Tulu Wikimedia community has organised.

30 Wikipedia editors participated and created about 89 new articles. Interestingly, 12 of these 30 participants crossed more than 10 edits. Some of the new participants faced problems with using the input methods and the Wiki-markup. It is important to note that because of the lack of addition of Tulu-alphabet code points in the Unicode chart, and the speakers being well conversant in Kannada, they are using Kannada script for Tulu Wikipedia Incubator project. There is a plan to organise a monthly meetup and/or edit-a-thon to continue the momentum these editors have brought in. Tulu Sahitya Academy has kindly supported the event.

More details in the event page (in Tulu).

For more details visit https://cis-india.org/openness/wikipedia-edit-a-thon-in-mangalore-to-bring-tulu-wikipedia-live

Tulu Wikipedia edit-a-thon in Mangalore

pavanaja — 2016-06-18T18:06:28Z

This edit-a-thon is aimed to encourage more Tulu speakers to contribute to Tulu Wikipedia which is currently in incubator. The edit-a-thon is being organised by the Tulu Wikimedia community in Mangalore.

For more details visit https://cis-india.org/openness/events/tulu-wikipedia-edit-a-thon-in-mangalore

Kannada Wikipedia Workshop in Mangaluru (Coverage in Prajavani)

pavanaja — 2016-06-18T18:14:22Z

As part of its ongoing partnership with St. Aloysius College, Mangaluru, CIS-A2K along with the Wikipedia community in Karnataka organized a workshop on Kannada Wikipedia. Prajavani reported this in their edition published on July 5, 2015.

For more details visit https://cis-india.org/openness/news/kannada-wikipedia-workshop-july-5-2015-coverage-in-prajavani

Kannada Wikipedia Workshop in Mangaluru (Coverage in Udayavani)

pavanaja — 2015-09-13T06:17:49Z

As part of its ongoing partnership with St Aloysius College Mangaluru, CIS-A2K co-organized a workshop with the Kannada wikipedia community on Kannada Wikipedia. Udayavani covered this in their edition on July 5.

For more details visit https://cis-india.org/openness/news/kannada-wikipedia-workshop-july-4-2015-coverage-in-udayavani

Upload More Kannada Articles on Wikipedia

pavanaja — 2015-09-13T06:09:34Z

Uploading information in Wikipedia helps to develop language, said Indian Languages Programme Manager U B Pavanaja here on Saturday. The article was published in Indian Express (Mangaluru edition) on July 5, 2015.


Above: A scanned version of the article that appeared in Indian Express on July 5, 2015.

For more details visit https://cis-india.org/openness/news/indian-express-july-5-2015-upload-more-kannada-articles-on-wikipedia