Materials Evaluation

Evaluation%20Policy%20Five-Year%20Report

Here’s a vocabulary exercise I found while browsing through material that Gerry Sweeny, a one time colleague at ESADE Idiomas, gave me.

Vocabulary in Context

The following sentences contain nonsense words. Can you make sense of them?

  1. The sentence was written on a piece of drurb.
  2. Most drurb, like snow, is osgrave.
  3. Cats are domestic ningles.
  4. Polar bears, which are osgrave ningles, live where there is cridlington.
  5. If you set fire to drurb, it firtles.
  6. If you pour narg on firtling drurb, the flames go out.
  7. If you put cridlington into hot narg, it frumes.
  8. Cridlington frumes at a bazoota over 0º C.
  9. Narg boobles at a bazoota of 100º C .
  10. We frize bazootas with a nast.

What do you think the nonsense words mean in the above sentences?

  1. drurb
  2. osgrave
  3. ningles
  4. cridlington
  5. firtles
  6. narg
  7. frumes
  8. bazoota
  9. boobles
  10. frize
  11. nast

 

CSLB-logo-400x126

I’m currently looking through material available to members of the Cooperativa de Serveis Linguistics de Barcelona, with the idea of getting a materials bank together which would help members to avoid using coursebooks.  While there’s an ambundance of ELT materials available online, it’s difficult to quickly find material that satisfies a few basic criteria, such as relevance, quality, useability and legality. Neill McMillan and I met recently and we reckon that we need to assemble a lot of material which satisfies these criteria, or rather, well-considered criteria that we can all agree on, and then classify them according to fields such as, off the top of my head, level, topic, media, grammar point, and skill. The idea is to give members access to a data base of materials where they can find written and spoken texts, with accompanying worksheets, at a certain level, topic, etc., so that they can easily confect everything from an ESP course with appropriate tasks, to lesson plans, to fillers. Maybe you’re only looking for a text; maybe you’re looking for a text plus worksheet, maybe you’re looking for a fresh aproach to practicing a function; maybe you need a good clear explanation of some grammar point, maybe you’re trying to get together a proposal for a 50 hour course aimed at auditors, and so on.  I should add that I have a particular interest in developing a process syllabus, which I’ve discussed in a previous post and which relies on a materials bank.

untitled

So we see the challenges of this project as being to decide on the criteria for any bit of material, to decide on how the collection of the individual bits of material is organised in the data base, and to indicate links among them.

Looking at the worksheet above, what to do? Supposing that it were well presented, and that there were no copyright issues, does it warrant inclusion? Is its openness a good thing (allowing teachers to exploit it in their own way), or does it need some lead in and some further work? Is it useful, anyway?  More generally, how do we judge it’s worth? If you look at most of the literature on materials evaluation, you’ll be hard put to apply the frameworks to this, because most frameworks are, either explicitly or implicitly geared to coursebooks. Rather than indulge in a rant, I invite you to give your opinion. If you were getting a materials bank together, would you include this?

13 thoughts on “Materials Evaluation

  1. I’d suggest that the activity is probably too hard for anyone it would be useful for. People who need a crash course in picking up new vocab from context would find the contexts here baffling, and students capable of picking up “temperature” from “at a ____ of 100 degrees” probably don’t need the help. I have a hard time picturing this activity working as is with any of the learners I’ve seen.

    Liked by 1 person

    1. Hi Mark (Is that right?),

      Thanks for your comment. I personally don’t think it’s that hard, but I’ve got no evidence. But anyway, what about the idea behind the exercise? If the exercise were “easier” would it be something you’d use?

      Like

      1. Yup, that’s me. I like the idea of the exercise as a semi-explicit treatment of how one can hypothesize meanings of unfamiliar words when doing extensive reading. I work (or worked) in a context that was rather hostile to the process of hypothesis-building and -testing and I would recommend something like this as an introduction to the idea that unknown words you come across are not JUST evidence of one’s educational failings. Keeping in mind that if extensive reading is being planned properly they won’t be guessing one or two words in every sentence.

        Like

  2. Hi there,
    One principle to stick to, for me at least, would be to avoid using learning time for “getting off track” materials or activities. Well, how to define “the track” or the shortest distance between learning effort and accomplishing ones communicative aims? Working towards a definition, this track would be a mediated road map informed by needs analysis, student profile, and teaching expertise.

    The latter should allow a teacher to maneuvre through the maze of nice looking activities. This imaginary (best, or optimal) track would avoid activities that make students take detours as in having their memory capacity loaded with items other than the language to be learned. Taking another concept, an extended focus on form approach should prune away all unnecessary information as the term form implies a direct reference to concrete language.

    The activity posted here is attractive as brain teaser for people that like to work out language puzzles (language teachers producing activities attractive for language teachers). But I am not sure what the benefit for the language learner would be. Make them alert of parts of speech; increase their understanding of English syntax? The difficulty of the discussion, what do the words mean, would call for advanced levels only.

    Regards,

    Thom

    Like

    1. Hi Thom,

      Thanks for your comment. It sounds a bit like we know the “track” better than we do, but it’s, as always, an interesting opinion.

      Like

      1. Dear Geoff,
        Yes, I know. Using “…” is not enough to soften the edges. I think this is part of our struggle, to get as close as possible to effective language teaching; there is such an abundance of teaching material out there that the ability to sort the chaff from the wheat becomes that much more important.

        (I am not sure if I commented here or elsewhere,… at times one comes across the notion that SLA research does not speak to materials development or methodology / pedagogy. I dislike this position. Maybe I am naive in hoping that SLA serves as feeder science for ELT and findings in first and second language acquisition do indeed, or should, inform our decisions in the classroom. I think if one cannot harmonize SLA “principles” (here they are again) with teaching practice, one is adrift in too many nice activities students will like. Of course, one has to live with preliminary principles, and in the end live at best with a deliberate choice. So, yes, any kind of track is subject to debate. I think we, as human species, are quite good at that. We advance with error until we know better.)

        Regards,

        Thom

        Like

  3. Hi Geoff.

    I’m not sure the exercise is particularly unsuitable or not: one of the good things is you are sorting out resources for a specific context that may be useful for others as opposed to sorting out a bunch of resources suitable for everyone but necessary for one local context.

    So, it begs the questions: why would you choose to use it? Is there better material available? Is it likely to engage the learners (current and/or future) or at least provide benefits in their learning? If you can justify it (and you know Spanish/Catalan learners better than me), you tag it and keep it; if not, it falls by the wayside.

    Like

    1. Hi Marc,

      Good points – thanks. One of the questions is certainly “Is there better material available?”, and maybe we should give materials that we’re considering for inclusion in the materials bank a rating out of 10, or something like that. The question of its relevance for Spanish/Catalan learners is also pertinent, and maybe here we need some way to indicate how “global” or “local” the material is.

      Liked by 1 person

  4. Hi Geoff,

    It sounds like a very interesting project – the factors to be considered when choosing which materials to include seem vast. On a side note, I wonder if teachers themselves could be seen as “materials banks” – that when we are in staff rooms together and someone needs a reading text on a certain subject for a certain level, a colleague might have (or remember) a particular text that might be suitable. We all store up this stuff as we go through our teaching careers, like chefs and recipes. Makes me think that perhaps a valuable contribution (to our community and projects like your own) could be for those of us messing around on Twitter to simply share a text of some sort, the context in which we used it and perhaps a short note about it.

    By the way, I very much enjoyed the exercise at the start of your post. But I think I enjoyed it in that teacher-ish way (oh great, a chance to show how smart I am!). I think as a language learner, I’d groan if asked to do this.

    All the best,
    Stephen

    Liked by 1 person

  5. Thanks for raising this very thorny issue – in fact its very thorniness is probably why we haven’t addressed it much in the design of our materials bank thus far. The range of possible criteria is so vast. But maybe we could “booble” it down a bit.

    Taking your example, my first instinct is to ask “What is this practising/studying?” and “Why?”. Firstly it seems it’s either to review a set of vocab or/and to practise deducing meaning from context. The second question is unanswerable – although it may lie in the preceding bits of material this is cribbed from. And this for me counts against the inclusion of this in our materials bank. It’s a disembodied exercise. It would only work as a time filler, as has been mentioned in the comments. In terms of vocab review, it would be a miracle if the particular group a teacher has at any given time would need to review precisely these items. And if our focus is on deducing meaning, we might ask if there were not a better way of doing this, e.g.:

    – in a more coherent/cohesive text such as those our students might need/want to read and need to guess at some key unknown vocab
    – with real lower-frequency words rather than nonsense words (e.g. “parchment” for paper, “simmer” for boil), or –
    – just an authentic text of interest to Ls where the T/Ls pick out some vocab for Ls try to guess the meaning of

    Maybe we’d decide that the concept of the exercise as a vocab review is sound in itself, so we present it as an activity framework – “here’s a way of spicing up a vocab review gapfill. To be incorporated into any more full-blooded, contextualised lesson based on (a) specific communicative objective(s)”.

    So then from this maybe we have some criteria to start off with:

    – how effectively does it practise or study what it sets out to practise or study (within the constraints of level, local context) (I understand that “effective” needs to be clarified and thought through carefully – may include aspects such as doability, enjoyability, authenticity, utility in developing memory, accuracy, fluency, subskills etc.)
    – if it’s not effective, can it be tweaked to make it more so;
    – is it to be done within a context/lesson with clear communicative or language goals (again appropriate for the level/locality, but there could be a wide range of acceptable goals if our aim is for Ts to select materials which fit with their Ss´ specific needs)
    – if not, can it be presented as an activity framework easily adapted to a wide range of possible contexts/goals

    There, I’ve stuck my neck out – hack away!

    Like

  6. Hi Geoff,

    First, as a developing critical practitioner working in English Teacher Education, I must thank you for this excellent blog.

    Personally, I don’t think I would include the exercise in a materials bank as I struggle to see how it could be meaningfully exploited by teachers and see many ways in which it could be ‘mis-used’: “of course ‘singles’ means animal” – The danger of a single story, what about: feline, creature, -ated, etc.?

    There are certainly too many easily accessible bad materials so the creation of a materials bank seems very pertinent, I would look for materials which are purposefully generic with concrete possibilities for ‘localization’ encouraging teachers to bridge the distance between their students’ contexts and the materials used. I would also not shy away from including grammar-based activities. Finally, it would seem that the myth of the ‘communicative classroom’ is falling away, I feel that we as teachers need to develop high-quality, meaningful grammatical materials which will foster a meaningful process in our students.

    Peter

    Like

Leave a comment