Locked History Actions

Diff for "Spinn3rDataSet"

Differences between revisions 6 and 7
Revision 6 as of 2014-09-06 01:21:32
Size: 1084
Editor: NikoColneric
Comment:
Revision 7 as of 2014-09-06 01:22:46
Size: 1090
Editor: NikoColneric
Comment:
Deletions are marked like this. Additions are marked like this.
Line 16: Line 16:
[[Spinn3rFormat]] - provides detailed description of version transitions and parsing.  * [[Spinn3rFormat]] - provides detailed description of version transitions and parsing.
Line 18: Line 18:
[[Spinn3rDataSetArchive]] - more info about processign.  * [[Spinn3rDataSetArchive]] - more info about processign.

Spinn3r data set on Hadoop cluster

This page provides all informaton about Spinn3r data set stored on Hadoop cluster.

Data records versions

There are several verison in which the records are stored. It is important to know which version are you processings, since depending on the version you know what fields are available for this record and how the text was preprocessed. For example, in some versions there is no capital letters, no raw html fileds, etc.

From

To

Version

2008-08-01

2010-07-13

A

2010-07-14

2010-07-26

B

2010-07-27

2013-04-30

C

2013-05-01

2014-05-30

D

2014-06-01

...

E

More information