DESCRIPTION

FEATURES

In addition to any one of the actions, data may be further processed in that fields may be filtered (keepfields or rejectfields).

The result may replace the original data set in memory or be kept in memory along with the original data (stack). In either case, after proc processdata is finished the result will be considered the "current data set" which all plotting procs will use.

You can write the results to a file (outfile) for processing by other programs, or to read back in and process further using proc getdata filter.

For additional processing capabilities see also proc getdata filter.

EXAMPLE

VARIABLES THAT ARE SET

NRECORDS = Number of rows in the data result.

NFIELDS = Number of fields per row in the data result.

TOTALS = If totals, percents, or accumulation are being done, this variable will be set to hold the field total(s). If more than one field is being operated on, this will be a comma-delimited list of totals; individual totals may be accessed in your script using something like the following, which would access the first total in the list: #set T = $nmember(1,@TOTALS)

BREAKFIELD1 .. n = Current contents of break fields, when action is breaks.

MANDATORY ATTRIBUTES

ATTRIBUTES

action a

The type of processing to perform. Often the fields attribute is used to indicate which field(s) are involved. Legal values for a include:

accumulate Rewrite field as a cumulative series (accumulation). The field(s) to operate on must be given in the fields attribute. For example, the data set on the left would be transformed to the one on the right (fields: 2):

	A21 3			A21 3
	A22 5		-->	A22 8 
	A23 2			A23 10
	A24 1			A24 11

breaks (new in version 1.39). Break processing. In data processing terminology, "break processing" is the act of passing through a sorted data set and taking some action when a key field or fields change. Break processing is significantly more efficient than scanning a data set multiple times with a select statement. The data set must be sorted such that key fields are grouped. The key field(s) must be specified using the fields attribute. The BREAKFIELD1 .. n variable(s) will be set to the current contents of the break field(s). Your script can detect when the entire data set has been processed by checking the NRECORDS variable (equal to 0), or the BREAKFIELD1 variable ($strlen of 0). A gallery example that uses this feature is mouse. See MORE ON BREAK PROCESSING below for more information.

breakreset Reset the "current row" to the beginning of the data set, for the occasional time when more than one pass through a data set will be done using the breaks action.

count Collapse data by counting the number of instances of a key field. Input data must be sorted (or at least grouped) on the key field. Resulting data set will always have two fields. One or two fields must be specified using the fields attribute. If one field is specified, the result fields will be 1) key field, 2) count. For example (fields: 1):

	062698	 		062698 2
	062698 		  -->	062898 1
	062898 			070198 3
	070198 			070498 1
	070198	 
	070198
	070498 

If two fields are specified, the result fields will be 1) key field, 2) sum of the numeric contents found in the second specified field. For example (fields: 1 2):

	062698 4		062698 10
	062698 6	  -->	062898 3
	062898 3		070198 9
	070198 2		070498 2
	070198 4
	070198 3
	070498 2

See also the gallery example hitcount

percent Rewrite one or more fields as percentages of its field (column) total. The field(s) to operate on must be given in the fields attribute. For example (fields: 1):

	8		40
	4	   -->	20
	3		15
	5		25

reverse The last record becomes the first one; the record order is reversed. For example (fields: 2):

	AXB 34		DIF 14
	BYA 22	   -->	CES 52
	CES 52		BYA 22
	DIF 14		AXB 34 

rotate First row becomes 1st field, 2nd row becomes 2nd field, and so on. This may be useful in that most of the plotting procs work from data fields, but sometimes data is given (or is more intuitive) in rows. For example:

	A 2 4 6 8 10	-->	A B
	B 3 6 9 12 15		2 3
				4 6
				6 9
				8 12
				10 15

select Select certain records. Result will be made up of records meeting the condition given in the select attribute.

total Compute field total(s) only and place total(s) into the variable TOTALS (see above). This action does not rewrite data. The field(s) to operate on must be given in the fields attribute.
The decimal format of the total(s) is controlled by the resultformat attribute. If total(s) are to be written in presentable notation (a spacer for thousands, etc.) the resultformat attribute may be preceded by a n, e.g. n%7.0f.

fields dfield list

The field(s) to be operated on. Required for any action that involves data fields.
Example: fields: 2 5 6 7

keepfields dfield list

If specified, only the listed fields in the original data set will be kept. The others will be rejected. action may be anything.
Example: keepfields: 4 5 6

rejectfields dfield list

If specified, the listed fields in the original data set will not be kept. The others will be. action may be anything.
Example: rejectfields: 1 2 3 4

fieldnames namelist

If specified, the names given in namelist may be used in any plotting proc to identify data fields. namelist is a space- or comma- delimited list of names. Names may include any alphanumeric characters with a maximum length of 38, and are case-insensitive.
Note that if field names are specified in proc getdata and then proc processdata is used to alter the order of fields or delete fields, then this fieldnames attribute must be used in order to redefine the field names properly.
Example: fieldnames: date group n

resultformat printf-spec

Controls the decimal format of rewritten percents, accumulations, totals.

keepall yes | no

If yes, original fields are preserved when doing accumulate, percent, or total. Thus the result will have more fields than the original data set. Default is no.

select selection-expression

Used when action is select in order to specify the selection condition.

stack yes | no

The default is yes which causes the result data set to be "stacked", and the original data set is preserved in memory. If no, the result data set replaces the original data set in memory.

showresults yes | no

If yes the data are shown after processing, as a diagnostic aid.

outfile filename

If specified, results are written to filename, in tab-delimited format, for processing by other programs, or to read back in and process further using proc getdata filter. This only works with stack: no

MORE ON BREAK PROCESSING

The breaks action allows break processing. In data processing terminology, "break processing" is the act of passing through a sorted data set and taking some action when a key field or fields change. For example, if we were processing a list of charges ordered by paying budget number, we could use a break processing strategy to pause and generate a statement for one budget number, when we reached the point in the data set where the budget numbers changed. Then we would continue on. The breaks action allows a similar thing to be done with plotting. Break processing is significantly more efficient than scanning the entire data set multiple times with a select statement, especially with larger data sets.

The data set must be sorted such that key fields are grouped. The key field(s) must be specified in the fields attribute. proc processdata is generally called within a #loop. When proc processdata finishes, the "current data set" will be the block of data from the previous break to the current break. Subsequent invocations of proc processdata will continue from the previous location in the data set. The original data set is always assumed (i.e. it is not necessary to use proc usedata).

Your script can access the current contents of the break field(s) via the BREAKFIELD1 .. n variable(s). Your script can detect when the entire data set has been processed by checking the NRECORDS variable (equal to 0), or the BREAKFIELD1 variable ($strlen of 0). The following is an example:

 	#loop
 	   #proc processdata
 	     action: breaks
 	     fields: 1 2 3
 	   #proc endproc
 
 	   #if @NRECORDS = 0
 	     #break
 	   #endif
 
 	   #proc page
 	   title: Account @BREAKFIELD1
 
 	   #proc bars
 	   ...
 
 	 #endloop

Limits: up to 5 break fields may be used. Comparisons for equality are limited to the first 50 characters.