Research directly comparing different health-related interventions is essential to confirm that the goals of those interventions are achieved (Horn & Gassaway 2007; Jutai et al. 2005). Comparative effectiveness research for wheelchairs intended for use in resource-limited environments is needed (Borg & Khasnabis 2008; WHO 2011).

The ability to move over different surfaces encountered in daily routine is a key aspect of wheelchair effectiveness (Kirby 2011; Mortenson, Miller & Auger 2008). Wheelchair users and family members have often placed highest priority on mobility and the psychosocial benefits that result from it (Hosseini et al. 2012). In fact, wheelchairs are provided specifically to those people who are unable to get around in their daily environment effectively by walking. Therefore, comparative effectiveness studies on the mobility facilitated by different wheelchair types on commonly encountered rolling environments are needed (American Medical Association 1994; Borg & Khasnabis 2008; Matter et al. 2016).

Wheelchairs intended for use in low-resource areas should enable good mobility on commonly encountered rolling surfaces. Wheelchairs designed for use in the United States and Europe may not provide adequate mobility because of the differences in the environments of daily life. In developed areas, populations spend more time indoors on smooth surfaces than they typically do in low-resource areas; therefore there is more need for rough terrain wheelchairs for low-resource areas (Winter et al. 2010). Ramps and wheelchair friendly public transportation are also more likely to be available in wealthier regions (Borg, Lindström & Larsson 2011b). In contrast, wheelchair users in low-resource areas often spend considerable time outdoors where they and their wheelchairs encounter rough terrain, and where public transportation is difficult (Blanford et al. 2012; Borg et al. 2011b; Monk & Wee 2008; Sietchiping, Permezel & Ngomsi 2012). In low-resource areas, living spaces such as houses and schools may be small and sometimes crowded; therefore, the ability to move a wheelchair through tight spaces is also crucial; building codes may not require ramps and other modifications that make wheelchair access easy; thus, coping with curbs is necessary (Borg et al. 2011b; Matter et al. 2016). The environment is so different at some locations that donated wheelchairs from more developed settings have been found to be nearly useless (Mukherjee & Samanta 2005). Wheelchair centre of gravity, caster and wheel sizes, wheelbase length and other design characteristics may impact the ease of movement in different rolling environments quite differently (Cowan et al. 2009). Characteristics of casters and tires including the quality, texture, firmness and condition of the bearings and wheels also impact ease of rolling (Frank & Abel 1989; Kauzlarich & Thacker 1985; VanderWiel et al. 2016).

Organisations manufacturing and providing wheelchairs for low-resource areas face tight financial constraints that are addressed by different organisations in different ways (USAID/WHO 2012). Often, wheelchair availability in low-resource areas may limit the ability of therapists to ideally fit wheelchair users. For example, active users may be put into a range of more or less ideally appropriate wheelchairs depending on what is available (Gartz et al. 2016). The type of wheelchair provided often simply depends on which type is available when one is needed. Studies carried out in the environments where the wheelchairs are used are essential to provide feedback to maximise effective use of limited funds (Jutai et al. 2005).

One of the challenges of comparative studies is the great variation in capability among wheelchair users (Hoenig, Giacobbi & Levy 2007; May 1997; Mortenson et al. 2008). Variations among wheelchair users in skill and capability level can mask the negative impact of a poor wheelchair design. Unless a protocol is used in a repeated measures study design in which the user completes a test in one wheelchair and then another, the impact of wheelchair design on mobility may not be apparent (Hoenig et al. 2015; Mortenson et al. 2008; Rispin & Wee 2015). This type of repeated measures study protocol minimises the effect of individual variation because a participant is compared only with himself or herself (Coutinho, Neto & Beraldo 2014; Neto, Coutinho & Beraldo 2014; Walpole et al. 2006).

Even in a repeated measures protocol, a strongly skilled wheelchair user may be able to roll in most environments in almost any wheelchair; therefore, the ability to discern the effect of different wheelchair designs requires high discriminatory validity. Study design impacts discriminatory validity in several ways. Objectively measured data such as velocity may be considerably lower in one chair type than another when rolling on rough ground, but a study design which only uses a limited categorical response or completion score would indicate the task was completed successfully in both chairs. Objectively measured physical performance data such as velocity and heart rate also minimise variation because of raters’ perceptive frames (May 1997). In addition, unlike categorical data, objectively measured data is often suitable for powerful parametric statistical tools such as analysis of variance (ANOVA) (May 1997; Walpole et al. 2006). The length of a timed test impacts discriminatory power with longer tests amplifying differences in velocity (Kosak & Smith 2005). Longer timed roll tests are also more likely than shorter tests to discern differences. A change in the ease of rolling which impacts energy cost is more evident as participants move into aerobic exercise about 2 min into a test (Berne et al. 2004; Kosak & Smith 2005).

Direct questionnaire feedback from wheelchair users provides insights on capability and mobility not available any other way (Neale & Strang 2015; Reeve et al. 2013). Questionnaire design can enhance or reduce the ability to discriminate differences (May 1997; Reips & Funke 2008). Visual analogue scale (VAS) question format produces continuous data which has been considered suitable for parametric statistical analysis (Philip 1990; Walpole et al. 2006). Qualitative comments directly provide the wheelchair user’s understanding of the reasons for ease or difficulty (Neale & Strang 2015). Discriminatory validity is enhanced by a mixed methods protocol which includes qualitative data, allowing wider scope of understanding and triangulation (Fielding 2012). The Quebec User Evaluation of Satisfaction with Assistive Technology obtains user feedback on their satisfaction with any assistive device (Demers et al. 2002). It can and has been used in comparative effectiveness studies assessing user satisfaction with assistive technology (Deems-Dluhy et al. 2016; Sadiya, Pattnaik & Mohanty 2016). However, it is not a physical performance measure and does not supply information specific to wheelchair mobility.

Questionnaires aimed at assessing individual capability and physical performance include Functional Independence Measure and the Barthel Index (Kumar et al. 2013; Ottenbacher et al. 1996; Wade & Collin 1988). Wheelchair-specific questionnaires include the Wheelchair Skills Test Questionnaire and Functioning Every Day in a Wheelchair questionnaire (Kirby et al. 2004; Mills et al. 2002).

Directly measured or observed physical performance measures intended to assess a wheelchair user’s capabilities are often called skills tests (Kirby 2011; Oyster et al. 2012). They include the Wheelchair Skills Test, the Wheelchair Propulsion Test, the Wheelchair Users Functional Assessment, the Wheelchair Circuit, the Obstacle Course Assessment of Wheelchair User Performance and the Wheelchair Physical Functional Performance (Askari et al. 2013; Cress et al. 2002; Fliess-Douer et al. 2010; Kirby et al. 2004; Mortenson, Miller & Miller-Pogar 2007; Routhier et al. 2004; Rushton et al. 2013; Stanley et al. 2003). The above measures are not primarily designed for use in repeated measures studies assessing differences in mobility because of wheelchair design. In spite of its focus on the evaluation of individual wheelchair user’s skills and capacity, the Wheelchair Skills Test has been used in studies comparing design changes in tilt in space and anti-tip devices. In the tilt in space study, the Quebec User Evaluation of Satisfaction with Assistive Technology and a VAS question regarding perceived exertion were also completed (Kirby et al. 2008a, 2008b).

Objective quantitative data on mobility have also been obtained in laboratory settings (Askari et al. 2013; Coutinho et al. 2014; Cowan et al. 2008; Cress et al. 2002; Neto et al. 2014; Yang et al. 2006). However, a laboratory setting does not perfectly mimic the conditions of daily use. It is important that effectiveness studies are also done in the environment where wheelchairs are used (Jutai et al. 2005; Toro et al. 2012; WHO 2015).

The Aspects of Wheelchair Mobility Protocol (AWMP) was developed to be used in low-resource areas in a repeated measures format to discern differences in mobility because of wheelchair design (Rispin & Wee 2015). There is a tension between every rolling environment that may be of interest, and keeping a protocol short, simple and usable. Because smooth and rough surfaces, tight spaces and low curbs are commonly encountered, and because each interacts somewhat differently with the wheelchair design, these surfaces were included in the AWMP (Rispin & Wee 2015). Face validity is the logical subjective expectation that a protocol will test its target construct (Jerosch-Herold 2005). To that end, mobility was directly measured using timed roll tests similar to the long validated timed walk test protocols (Enright 2003; Rispin & Wee 2015).

Methods were selected with the intention that quantitative data would be continuous and suitable for powerful parametric statistical tools such as ANOVA (Rispin & Wee 2015; Walpole et al. 2006). Performance tests were of sufficient duration to include the transition to aerobic respiration (Neto et al. 2014; Rispin & Wee 2015). Visual analogue scale responses and comments were solicited from participants, and exercise and resting heart rate were monitored using research grade heart rate monitors (Crapo et al. 2002; Rispin & Wee 2015). The format of the participant response questions in AWMP was based on that used in the Lower Limb Function Questionnaire. This format was selected because VAS format is very likely to provide data suitable for parametric statistical analysis tools, and because each question includes an opportunity to write a comment providing qualitative explanatory information (Funk et al. 2016).

Measurements of heart rate or oxygen consumption provide continuous objective data on the energy cost of movement (Coutinho et al. 2014; Neto et al. 2014). In earlier iterations of the AWMP, the physiological cost index (PCI) was calculated rather than directly comparing exercise heart rate (Rispin & Wee 2015). Slowing down when encountering difficulty or awkwardness is the strategy that has enabled timed walk tests to be validated by measuring difficulty in ambulation (Holland et al. 2014). However, slowing down is not the only response to a greater difficulty in moving. Continuing at the same pace and spending, more energy per unit time is the strategy of dealing with difficulty that underlies monitoring heart rate and oxygen consumption per unit time as a measure of difficulty of movement (Conger & Bassett 2011). PCI calculation includes velocity, and thus includes both strategies of dealing with difficulty; however, this makes it difficult to tell which strategy is most commonly used (Ijzerman & Nene 2002). Circadian rhythm, the cost of digestion, body temperature and other factors have a larger proportional impact on non-exercise heart rate than on exercise heart rate. Because PCI calculation includes non-exercise heart rate, these factors would likely impact PCI more than they would do the direct measurement of exercise heart rate (Berne et al. 2004).

The objective of this study was to obtain data that sheds light on the mobility provided by wheelchairs designed for low-resource areas as they roll on surfaces and in situations commonly encountered there. Our hypothesis was that an updated version of the AWMP which used exercise heart rate instead of PCI would have the discriminatory validity to provide comparative effectiveness data on four types of wheelchairs commonly provided to wheelchair users with strong upper bodies at our study site. Specifically, we hypothesised that ANOVA results for velocity, heart rate and participant response VAS scores would indicate significant differences between wheelchair types. Participant comments would shed light on the perceived reasons for ease or difficulty. Meaningful discriminatory validity would be confirmed by results which would enable wheelchair manufacturers to make responsive changes.

Discriminatory validity of the AWMP was confirmed through ANOVA analysis which discerned significant differences between wheelchair types for velocity and participant response scores. The numbers and types of comment topics explained ANOVA results and added other qualitative information. Results from this study led to responsive design change confirming the value of the AWMP in providing comparative effectiveness data.

We had decided to use mean exercise heart rate in analysis instead of PCI for two reasons. We wanted to differentiate between two strategies of dealing with increased physiological difficulty, and reduce the effect of other factors that impact basal metabolic rate. Although mean exercise heart rate differentiated between rolling environments, we could see clearly that exercise heart rate, a measure of energy cost per minute, did not significantly differ between wheelchair types on the rough, smooth and tight tracks. If a wheelchair was more difficult to use, participants seemed to simply slow down. The only test for which ANOVA indicated a significant difference in exercise heart rate was the curb track for the 12 strong and skilled participants able to complete this test in all four chair types. This significant difference was because of higher mean heart rate in the WRR chair than in the other types. However, the significantly higher heart rate in WRR did not seem to impact perceived difficulty much, because visual analogue scores for WRR were not significantly different from those of the MRT wheelchair (Figure 2), and score patterns paralleled velocity more closely than heart rate. This seems to indicate that participants’ perception of difficulty was more closely related to velocity than heart rate. It would seem that by dropping the use of heart rate monitors from the protocol, we could simplify the AWMP considerably without much loss of discriminatory validity.

Results of this study are specific to the conditions at our study site. Although it has much in common with other sites in low-resource areas, it is of course unique as are all locations. Tracks would not be identical at other study sites, and data from other study sites could not be directly compared with this study. For example, a rough ground track on an unpaved road at one location would not be exactly like a rough ground track at a different location. A low curb which is available and often encountered at one location might be 8 cm tall, whereas a curb at another location might be 10 cm tall. In studies carried out in North America, standardised rough surfaces have been developed and used (Sasaki & Rispin 2016). However, these standardised rough surfaces are large and not easily transported to field locations in low-resource areas. In addition, they would not perfectly model conditions encountered by wheelchair users. With the AWMP, study design, each participant is compared with themselves in their community location. AWMP could be used in many locations as long as there was no intention of considering the data exactly equivalent to data collected at another location. This flexibility is necessary in real-world research and is of key importance in studies carried out in low-resource areas. The use of letter grades as anchors for the VAS tends to standardise subject response and provide an understandable value to participants, researchers and manufactures (Funk et al. 2016; Rispin et al. 2017). However, the letter grades would need to be modified by whatever grading scale is in use in the culture in which a study is carried out.

Study wheelchairs were in the configuration most often utilised at our study site for those who can self-propel strongly. A broader comparison would have been provided if all four wheelchair types could have been included on all tracks, but this was not possible because of time constraints. Both the MRT and WRR chairs have the ability to be adjusted into less stable configurations with the centre of gravity closer to the rear axle, which, for skilled users, is thought to provide enhanced mobility (Motivation 2011; Whirlwind). Most wheelchairs at our study location had not been changed into this less stable but more energy efficient configuration. Therefore, we set the study WRR and MRT wheelchairs to a more stable configuration. Configurations of the same wheelchairs may routinely be set up differently at other study locations. However, additional studies at other locations are needed.

Results are specific to our study population of school age participants able to self-propel on rough surfaces. This population is not typical of the global population of wheelchair users, especially because obesity and age-related disability is becoming more prevalent in low-resource areas (WHO 2015). Validation is needed for the AWMP for other populations and cultures. However, it does seem likely that a wheelchair which is challenging for strong adolescent users will be even more challenging for older and more disabled users.

Wheelchair users were selected based on their identification by local therapists as having size and disability characteristics appropriate for the study wheelchairs. However, wheelchairs were not finely adjusted to each participant. Wheelchairs were of course not identical, and would have fit different participants somewhat differently. For each individual, this would impact results because appropriate seating is known to impact mobility (Borg, Larsson & Östergren 2011a). In this study, the sample size of 30 participants and the repeated measures format would have reduced the impact of the individual variation, unless the seating system of a wheelchair type was less suitable for most participants. In that case, the results were still likely of interest because the population of users with the types of wheelchairs in our study at our study location had the characteristics of our participant population. Results can be generalised only to the extent that other study sites may resemble our study site.

Wheelchair or user centre of gravity was not measured for each wheelchair and would have shifted somewhat for users of differing individual morphologies and resultant personal centres of gravity. However, body structure of individuals that resulted in a shifted centre of gravity would have remained the same for that individual in all chairs, so the within subjects protocol should have minimised the impact on comparative results.

All wheelchairs except HKC had inflatable tires. Before each data collection session, tire pressure was informally checked by pinching the tire, but was not formally checked using a tire gauge. A tire gradually losing air may not have been noticed and could have impacted results. In future studies, it would be wise to formally check tire pressure before each run.

The weather and other differences between days could have impacted our data. We chose to do each chair type on a different day because we felt that the fatigue which would result from attempting to do multiple wheelchairs on one day would have had a greater impact than any difference because of variations between days.

Accommodation to unfamiliar wheelchair types may also have impacted our data. Participants utilised multiple wheelchair types that included wheelchair types they had never used. A short accommodation period was included in the protocol; however, accommodation may take longer than a few minutes. If accommodation had affected results, FG2 results in this study could have been negatively impacted. A portion of participants were long-term users of each of the types of study wheelchairs. More participants were HKC users than any other chair, and yet all other study chairs outperformed the HKC chair. FG2 had the fewest long-term users.

The good discriminatory validity of the AWMP enables its use in comparative effectiveness studies that can provide much needed feedback enabling wheelchair manufacturers to optimise wheelchair design. Organisations that design and manufacture wheelchairs intended for use in low-resource settings are almost always not-for-profit and have a strong commitment to meeting the overwhelming global need for wheelchairs. Designs that hinder users’ abilities to roll forward on commonly encountered rolling environments limit the positive impact of wheelchair provision. The AWMP can be used in comparative effectiveness studies for other wheelchair types to inform beneficial design change. If AWMP studies are performed with locally available wheelchairs in locations where wheelchair users live, results could enable informed choices for wheelchair provision. Other stakeholders such as granting or charitable agencies providing funding could also benefit from comparative effectiveness data to inform wheelchair selection choices for different locations.