Thursday, October 30, 2014

False Negatives: Evaluations of Functionally "Nonverbal" Children

My daughter Maya, like many children with special needs, has undergone (too) many evaluations. Her skills and knowledge have been quantified, over and over again, with varying degrees of accuracy. Many parents of children with special needs speak about receiving the evaluation reports with a hint of PTSD in their voices and tears in their eyes. I see their posts in Facebook groups, about a son who has “the cognitive functioning of an 18 month old” or a daughter with “very little receptive language, according to the recent report.” Many have been devastated by these reports.

I am not one of these parents. I have not been devastated by the reports. 

Ok, the first one briefly knocked the wind out of me, but I recovered quickly.

Why not? Perhaps because Maya’s scores have been promising? Or “average”? Or “low average”?

No. No, not because of that. As a matter of fact, her first round of cognitive testing was abysmal. She scored in the bottom 0.4th percentile---meaning that 99.6% of all children her age were more cognitively advanced than Maya. Her receptive language evaluation (at 2.5 years old) estimated that she could produce 1-2 words and that she understood 1-2 words.

They said that she understood 1-2 words. At 2.5 years old.

I wasn’t gutted by the reports because I knew they were worthless. At the time, I thought everyone knew they were worthless---that they were just a means to an end (the end goal being to score “disabled enough” to qualify for services). I thought the results were like She’s in the 0.4th percentile—wink, wink. It wasn’t until I became more immersed in the special needs community, particularly the community of parents who have children who are functionally nonverbal, that I realized something startling---people are believing these numbers.

Parents, you cannot believe these numbers.

Not only will believing the numbers send you down some sort of spiral-of-terrible-feelings, but believing them will change your expectations for your child. The numbers will change what you believe your child is capable of, they will plant seeds of poisonous doubt, and they will corrode your ability to presume competence. If you have a child who doesn’t speak, one of your biggest, constant jobs in life will be to advocate for their people to believe in them . . . so if you start to lower your expectations, others will follow.

Plus, really, the numbers are garbage.

My undergrad degree was in science (zoology), my first master’s degree was in teaching secondary science, and I was a classroom teacher for 8 years. This means that I have both spent a fair amount of time reading scientific journals and that I have experience with assessments of children (and their limitations). I listen to news stories with a skeptic’s ear (who funded that study, what was the sample size, how did they control for these 12 other variables, etc). I see the limitations of a standardized test as quickly as I see the potential data collected. I take all reports with a grain of salt, by nature. But when I saw the way that Maya’s evaluations were done, I realized that it’s not just a grain of salt needed when looking at cognitive assessments (or receptive language assessments) of nonverbal kids, it’s a mountain.

Let’s  think about the tests: If an evaluation is being done on a child who can’t functionally speak, can’t read, and can’t write, answers to questions would basically be limited to two methods of production:  pointing to an image in a field (which item would you use to drink?) or completing performance tasks (such as arranging tiles to match a pattern). There are significant variables in both of these models that can result in scores that are erroneously low---false negatives.

pattern sheets that would be used with tiles

Performance Tasks: It’s easy to see why the performance tasks would result in low scores for many of our kids. The only examples that I clearly remember were about using tiles/blocks to duplicate or continue patterns. For a child like Maya, who struggles to use her fingers to manipulate anything small, this was a very challenging task. She also has apraxia, which means that the messages from her brain to her muscles get derailed---so she could be thinking "move the tile forward" but her hand won't respond. When we got to these questions, Maya refused to try and put her head down.

So, did her inability to engage with the task indicate that:  
            a) she didn’t recognize the pattern
            b) she didn’t understand the question
            c) she didn’t want to try a task that would be simultaneously cognitively taxing and physically                 close-to-impossible

Who knows. But for scoring purposes, that’s a fail. This is a false negative. The absence of a positive response in this situation doesn’t reliably indicate a cognitive limitation, but it will be scored as one.

Pointing at pictures: When I first saw a flipbook that would be used to evaluate receptive language, I thought it looked like a great idea. Maya was probably around 2 years old when I saw the test administered, and while I sat silently and didn’t interfere, I was surprised by how something that at first glance seems fairly straightforward and objective was actually very subjective and unreliable. There are a few things that need to be taken into consideration when thinking about these tests.

1. Personality: Children with complex communication needs often have a fairly passive personality with regards to getting their point across (due in large part to the fact that other people--parents, siblings, teachers--often step in to try to communicate for them). In addition to being passive, our kids learn that sometimes acting clueless is a great way to avoid the task at hand (hours of therapy reinforced this for Maya---act clueless, the therapist will adjust the task or change to a new one). While I can’t speak for all nonspeaking kids, Maya mastered “the blank stare” (in which she would simply stare around like she didn’t hear anyone and had no knowledge of what was transpiring around her) at a young age.  I will never forget watching a new therapist try to cheerfully get Maya’s attention to give her back her car keys: Maya! Maya, over here! Those keys are shiny, right?! I’m going to need the keys back now! Hey, Maya! and then I was all Maya, quit it and give her the keys and she looked up and handed them over. If a child stares blankly at an evaluator and doesn't engage with the test, that is not necessarily indicative of a lack of understanding. It's a false negative

2. Modified abilities lead to modified experiences: One page of the receptive language test had images of art related items (crayons, scissor, glue, etc) and Maya was asked to identify the scissors. Maya didn’t know what scissors were when she was two---she was an only child who spent all of her time at home with me and in home therapy, and she had basically no fine motor ability. She had never seen scissors. This is only one example, but there were several.  Did she know what scissors were? No. Was this indicative of her ability to understand, absorb, and recognize language? Of course not. It was indicative of the fact that her atypical abilities were leading her through an atypical childhood. It was a false negative. (I bet typical kids don’t know what chewy tubes or z-vibes are---Maya would have nailed those ones.)

3. A will, but not a way: For kids with disorders like apraxia or other neurological conditions, there are times when their body does not follow the directions being sent out by their brain. They may be thinking "reach out and touch the scissors, reach out and touch the scissors" but then see their arm reach forward and their hand make contact with the glue. If I stand up and quickly spin in place several times, no amount of me thinking "now run in a straight line!" is going to make that actually happen. It's a physical limitation. Modifications can be made to tests and testing environments to attempt to minimize these effects, but neurological motor planning troubles must be taken into account as a possible source of false negatives.

4. Communication vs. Testing: Children who are functionally nonverbal are often very interesting communicators who use a variety of methods to get their points across:gestures, sights, sounds, meaningful glances, avoidance. One thing that tends to be fairly consistent is the ability to communicate via pointing and pictures: even before Maya used picture cards or communication technology, she could communicate by pointing to the pictures in a book. She pointed to the cow, I would say “A cow! Mooooo!” She pointed to the moon, I would say “That’s the moon! It comes out at night.” Or, outside of books, if she pointed to the refrigerator, I would say “Are you hungry?”

During these tests our children are presented with a field of images and asked a question that has an “obvious answer”. The problem is that presenting a nonspeaking person with a field of images can be akin to saying “Check these out! Which one speaks to you? Which one reminds you of something? Which one do you want to talk about?”

 I'm willing to bet that if I showed this to Maya and said "Which picture shows the car in front of the house?" she would point to the first picture, look pointedly at me, and laugh. Translation: "Mom! There's a car driving into a house! This is ridiculous! . . . Wait, did you ask me a question?"

Example: Maya was 2(ish) and was in the middle of a receptive language test. The doctor flipped the page and said “Which one is the hairbrush?”, and Maya pointed to the toothbrush instead of the hairbrush. Obvious wrong answer, obvious confusing of one "brush" with another, right? No, not right. We had been talking about tooth brushing at home all morning. We had read a book about going to the dentist. We had bought a new toothbrush the day before. When the doctor flipped the page and asked about a hairbrush, Maya was already fixated on the toothbrush (which looked just like hers, by the way) and reaching toward it. As she tapped it, she turned to make eye contact with me. I saw her saying Look! A toothbrush, just like mine! We were just talking about brushing teeth! The doctor saw the wrong answer. A false negative.

Parents, take heart. Professionals, take heed. There is no reliable standardized way to assess the receptive language or cognitive function of a person with complex communication needs. Even now, with a robust language system, Maya has a way of seemingly jumping to unrelated topics that are actually related (but their relation is something that only she would know). Here’s a final example: On Tuesday Maya went to her after-school speech therapy, and this happened:

Therapist: “How was school?”
Maya: (glancing at the fire alarm over his head, in the corner of the room, and then using her device 
            to spell) “f-i-r-e.” (copied from the side of the alarm box)
Therapist: (looking up at the alarm) “Yes, that says ‘fire’ . . . but let’s focus-I asked you how your 
                 day was. Did you have a good day at school?”

There was no way for him to know that her school had a fire drill that morning.

He thought she wasn’t paying attention, but she actually was telling him about her day.


False negative.


28 comments:

Anonymous said...

Every word of this is so true! How can all those "specialists" be so clueless? I have been fighting this fight for my son for 20 years. I started by refusing any and all testing that would yield those "numbers" after the first really bad experience. "you can SEE that he is eligible for spec. ed. services, right?" Keep up the good work!

J said...

Love it Dana , I love the last example about the Fire Drill- just fantastic Go Maya !!

Casey said...

Thank you for this! Thank you for highlighting the motor differences in kids who are functionally non-verbal which lead to our schools' inability to presume competence. These "false negative" scores and related labels lead to demoralizing and restrictive educational plans filled with "behavioral therapies" and void academic opportunities. All too often these kids are cognitively intact but have no reliable way to fully communicate what they know. At Optimal Rhythms / ACCESS Academy, we are disproving the myths and providing the supports these students need to successfully learn in a general education envirinment (despite their erroneous psychological testing). Keep this discussion going!

Kerith Stull said...

I truly HATE tests for my 18yo nonverbal child. She uses an augmentative communication device at school but uses fairly fluent sign language at home -- yes, I know you're not a fan of sign but it truly works for her at home much more efficiently than her device. On top of her communication issues, she has fine motor issues, particularly on the right side. She has very little experience/success with manipulating puzzle pieces. Testers put together pieces from one test and another and another to get her IQ score. Her last number was 36. How can that be?? She functions at least at a fourth grade level academically. Talk about false negatives. Drives me crazy.

Unknown said...

This is an essential read for all that work with children with special needs. I have to say I feel that this is where the special educator is so important. Someone who works with your child on a daily basis is much better at assessing your child's abilities, not disabilities. The assessments we use like Brigance and SANDI are based on observations and direct assessment. Assessments in this manner are invaluable in helping set good educational goals for a child. I see how important rapport is to any assessment, its sad so much value is placed on outside evaluations for funding purposes. It drives me crazy that the school psychologist is in charge of triennial IEP meetings, when they have no relationship with the child and wind up presenting evaluations with "untestable" or low scores. I'm glad you've learned to not fixate on them and hope others will learn from you.

Dana said...

Kerith, I'm definitely a fan of sign for older children/adults who were raised using sign (and before the tech wave). And to clarify for possible new readers who are wondering why I don't like sign---I actually do, but only when it's used as one communication tool in a large toolkit. This post speaks more about that: http://niederfamily.blogspot.com/2014/05/the-limitations-of-sign-language-for.html

Unfortunately, MsSpeducate, I find that in house assessment are often equally unreliable. As you mention, rapport is SO huge---and if educators are starting with low expectations and not presuming competence they may not correctly interpret our kids . . . or, our kids may decide not to try for them. I've found taking video at home (of independent AAC use and of us engaged in literacy activities) to be invaluable. I share the videos with the team, and they have to believe.

Anonymous said...

My thoughts exactly. Thank you!

Anonymous said...

This is such a great post! I am a Speech Language Pathologist and have to give standardized tests in order to "prove" to the state or insurance that my clients have low enough language skills to receive my services. Many times, I know the test is not able to show what the client CAN do. I try to get this across in meetings with parents, but it's hard when you have to have numbers for the entity that will pay the bill. It's easy to get complacent about presenting these numbers and using them to develop therapy instead of using the information you gain from parents' knowledge and from working with the client. Thanks for the reminder. These tests are a means to an end (for the insurance company or the state) and should not be used to "get to know" the client.

Anonymous said...

WOW, it is as if you have lived in side my head! I have said much of this from day one, thank you for posting and reminding us all what is really important.

Anonymous said...

Thank you Thank you! As a mom I know all this. Thanks for putting it down on paper in such a well thought out, researched intelligent way.Another thing my son has been tested with is completely outdated material. "pint to the Clerk." Picture of a store and the cashier. I said to the tester who even heard of a clerk anymore? or mainstream media when your kid does not watch Disney, etc. etc. Or we live on a farm and there is tons of "city" stuff... buses and trains, etc. which he had barely ever seen. I say follow the money. The tests are money makers and why should the companies update them... that costs money. I have been so fristrated by the tests I refuse as many as I can(O here I am being the difficult parent). Thanks again.

Liz Tree

miriamp said...

Why don't people pay more attention? My first instinct with your therapist story would have been to ask about a fire drill! And I'm not the parent of a special needs kid or in the field. (My father is totally blind, so I did grow up with more of a sensitivity to "differently abled," but to at least ask if there had been a fire or a drill still seems like common sense to me!)

miriamp said...

Oh, now that I said my kids aren't special needs, some of them do/did have minor speech issues. (A lateral lisp on several sounds and a vowel shift.) I noticed this short of thing when I had my kids tested for that -- I had to sometimes interpret, because my child had answered the question right but the tester thought she had said "guy" instead of "girl." Or they used pictures my child was unfamiliar with and thus couldn't label. It was important that he know it was her pronunciation that was off, not her cognition, so we got the proper services.

LC said...

LOVE! I have two children who have had speech therapy through early intervention, so I've seen some of the assessment tools and YES! I can see how the issues would be magnified with a child with more significant language or cognition issues, but they exist even with kids who barely qualify for services. I like your original assessment of the assessment tests - simply a way to prove you need all the services available to be offered. and then as a parent, you tell the provider(s) what your child actually can and cannot currently do so they can set their expectations in line with reality.

Anonymous said...

So true.
As the parent of teens with disabilities, I can tell you that this system can wreak havoc on self-esteem. To be presumed to be not competent or unfocused, or to not be believed is tough to hear from "professionals" over and over again throughout childhood.

Anonymous said...

I am currently studying to be a Speech Language Pathologist, and I think it was really important for me to read your post. Throughout my practicums, I have actually found that assessments seem to be used too much, when they aren't necessary, appropriate or at all worthwhile. This has given me a lot to think about. Thank you!

anna v said...

Oh my gosh, thank you for posting this, this is exactly why every evaluation has been so frustrating and disheartening for me! You summed it up correctly, right on the mark. Thank you for sharing this!!

Katie said...

Thank you!! This would explain why my son always tests much higher in expressive language than receptive (how is that even possible?? He can speak it but he can't understand what he himself is speaking?)
Your examples are perfect.

Anonymous said...

What a wonderful post! I hit rock bottom when DS was 2. I took him to a new ped. to get a rash checked. Ped took 1 look at DS, said "I've seen kids like him, they rarely better". DS at the time was non verbal. I completely lost hope because here was a "professional" telling me this.

2 yrs later DS was seeing a child psychologist to see if he was getting the autism dx or not. The dr. was pointing out DS's deficiencies, meanwhile at 4 yrs old DS was READING the words in a book (& pronouncing them correctly)-aardvark, koala, flamingo, etc. As the dr was droning on I thought You.Are.An.Ass. Stop telling me what is "wrong" with my child, he's obviously bright & you are testing him verbally & giving him open ended questions which he cannot answer.

Anonymous said...

My daughter is an adult now, but was very much like Maya as far as testing is concerned,but with much worse fine motor skills. She had very good early experiences with a Psychologist over many visits from age almost 2 years to almost 6 when we moved out of state. That was primarily BECAUSE he was willing to presume competence and start with testing items for her age instead of following the test directions and start way below her age.
He also noticed that she almost always looked at the correct answer first and stared at it for a while, but then would refuse to answer or chose the incorrect answer. Because he knew her, he recorded BOTH her first answer and what she ultimately answered. The first answer put her as age appropriate as far as that test. He said her second answers were not even possible to get that many wrong by chance. His conclusion was that she was answering incorrectly on purpose.
When we moved, his written assessment included things like "she is a one trial learner. If you ask and she responds correctly, assume she knows and move on as you would for any non- disabled child....if doing testing, start AT or SLIGHTLY below her age level. If you start too low, you will lose her before seeing her true level."
Unfortunately, after we moved, they did none of those things. We refused Psych testing until we could not any more without losing services. Even though she was about 9, that school Psychologist said he "could not rely on" our previous Psychologist's testing because it was "done in a non-standard, unnormed method". He said he "could not" start at age appropriate test items because it would "not be valid". So, he began with age 9 month items, which she refused to answer. As a result, she tested at about 10 months on the test.
He also (I think, threw us a bone), by including a questionnaire of skills/abilities too. It included things like:
- Gross Motor skills: runs well, accurately throws a ball, catches a ball
- Fine motor skills: uses a scissor to cut accurately on a line, writes, colors in the lines, cuts food
- Self help: can prepare a simple meal, uses toilet independently, independent with hygiene (wash face, brush teeth)
- Safety: can safely play outside independently, can state name/address and phone number

Of course, as a no-verbal child with cerebral palsy who can't walk or stand and who has very poor fine motor skills, she could not do those things either

Anonymous said...

About pictures -
When I was a school nurse, one of the early elementary grade teachers came into the break room laughing. She had just finished a 'name the starting letter' exercise with her class.
She showed us a picture of a wig with bright red hair (like clown hair color).
She had shown the class the picture and asked who knew what letter it started with. Only one little boy volunteered and he answered, "w".
She asked him to tell the class what the picture was so everyone could hear the starting letter. He proudly said, "My mom made it for dinner last night. It's Wasagna".

I have to say that that the picture resembled lasagna about as much as a wig (and, as it turned out, none of the class knew what a wig was).

And naming things -
One little boy was shown some pictures and asked to point to the picture that showed a dad. His response was to ask "what's a dad?"
His family member was able to clarify that he called his dad "papa". When asked to point to the picture of a papa, he pointed to the correct picture. He had the same problem with "point to the picture of a purse" (the word his family used was "pocketbook".
Using the words he knew, he could point to the correct picture BUT, the items could not be scored because the standard instructions say to ask the child "point to the picture that shows a dad...point to the picture that shows a purse"
Many kids run into similar problems with identifying "easy" pictures like "Grandmother" and "Grandfather". If your grandmother is "Mee-Mee" and your grandfather is "Pop-pop" or some other chosen name, if you answer incorrectly, is it because you don't understand the concept or have never actually used that word to refer to that concept?

Suzanne Warburton said...

I totally agree. Edward was tested at 70 - 85 as a partially verbal asd 3 year old. I was shell shocked. This is the same boy you at the age of 2 1/2 could read words we had no idea he knew. He is now 4 1/2 and loves learning the periodic table, the Russian alphabet, the English Alphabet, numbers, the solar system, the rainbow, the skeleton, the world map and states and capitals of the USA (we live in New Zealand), writing (everywhere). He is between 70 and 85 IQ and I am the queens sister.

Suzanne Warburton said...

I totally agree. Edward was tested at 70 - 85 as a partially verbal asd 3 year old. I was shell shocked. This is the same boy you at the age of 2 1/2 could read words we had no idea he knew. He is now 4 1/2 and loves learning the periodic table, the Russian alphabet, the English Alphabet, numbers, the solar system, the rainbow, the skeleton, the world map and states and capitals of the USA (we live in New Zealand), writing (everywhere). He is between 70 and 85 IQ and I am the queens sister.

Meryllee said...

I realised at a young age that my child had a communication problem. Her first word was a loud firm NO!! Later I realised that she spoke in opposites. when the bath was too cold she asked for more cold water - isnstead of hot; she often said no when in fact she meant yes! There were many arguements and she was frequently in trouble as I seemed the only one who understood this. Eventually she got to conventional school after being delayed an extra y ear. She failed Sub A due to an inexperienced teacher who taught the class nothing for 6 months and was only replaced after the local Educational PSychologist in private practise assesed her - the 14th child in her class to consult him and realised there was a serious problem with the teacher at that school - but the damage was done and she repeated sub A. She was having occupational therapy - and speech therapy - her concept of time was absolutely upsidedown. A lot of time and effort went into helping her to stay in main stream, but in grade three at last she was accepted into a remedial school. She thrived in the small classes, and in grade 7 we applied to have her join mainstream school again. The Conners reports were all positive except the Maths teacher, who claimed she would never manage. We sent her anyway with the understanding that if she did not manage we could take her back to the remedial school. at the end of grade 10 we had her assessed for career guidance by a professor of psychology. The feedback we got was astounding - she had been underperforming......at MATHS!!! We changed her maths calss to higner grade and she thrived. Today she is in her twenties, and has a diploma in IT software programming. SHe will always have verbal skills problems of comprehansion and expression but has learned to cope.

Meryllee said...

It is just impossible to assess a person who annot read your question or understand you. Just because they are unable to respond to conventional tests they are labelled as below average, etc etc. Very sad

Anonymous said...

i understand the emotion behind this article but would like to see more facts that support the strong assumptions made about the value of standardized assessment tools and the assessors that give them.

Dana said...

Anonymous---tough request. Some facts are easy to find. Check out the normative/standardization data in the administrators' manual for the tests administered---many tests are used "off-label" for populations for which they haven't been standardized.

Another way to see the limits of the tests would be to give the primary caretaker a notepad in the corner while you administer it, with the instruction to write down every time they see an answer that is "wrong" but makes sense in another way (like the examples above).

Another way to see the limits of the tests would be to believe in the kids. If you've seen Maya's video or followed our story, you know that she was assessed in the bottom .4th percentile for cognition, and obtained a similar score for receptive language. Yet at age 8 she is reading on grade level. She is not an anomaly, or a miracle, she is a child whose parents could recognize the invalidity of the assessment tools and their use and chose to proceed by giving her a large communication system and presuming competence. I believe (and that belief has been confirmed by the sharing of numerous stories from other families) that she is an example of how standardized tests fail children with complex communication needs.

Unknown said...

I think this article is a good start. Testers have to know the child's easiest mode of responding (which is likely not using their hands) and likely use informal pictures and steadily gear up to a test plate. Jumping right in with an array of four is often way over a child's ability as a new participant in tasks to "find out what they know." It is a fair questions to seek "what they know" and this article points out some of the challenges in the "response mode" and possible competing factors pictures represent (expressive pictures or receptive pictures.) Perhaps the child deserves to be given the direction in a "meta" way so that they realize that it is a test? Lots of interesting points. When I "test," I take it extremely slow, starting with an array of 1, then an array of 2 with one blank, figuring out the most easiest response mode. I even separate out language and look at matching, and go through a simple "find the star" to see the best array and type of pictures. Receptive abilities can easily be over or under-estimated and it is fair to wonder what a child knows. Thank you for this thoughtful article.

Laura R. said...

The false negative thing is so familiar to me. I went through a few years in "gifted" classes, but in preschool and kindergarten I had failed several "intelligence" tests. Why? I have severe food allergies and chemical sensitivities. There were all manner of things with which I was not familiar, many common types of foods included. Why would I know what was missing from an ice cream sundae (the cherry, apparently) when I'd never had one or been around one?! At one point, while having a reaction and sensory overload, I had a doctor suggest I was nonverbal-autistic because I didn't answer his questions. This is an important article for teachers, not just testers. A child may know something but not be able to express it the same way another child might...