Case Study: Karuta at New Brunswick Theological Seminary

New Brunswick Theological Seminary (NBTS) is using Karuta as a significant tool for its assessment of students and of itself.  According to Dr. Terry Ann Smith, Associate Dean of Institutional Assessment, NBTS needed a tool that could give students a way to demonstrate their competency and that could provide the institution with a means of evaluating its programs.  Four years ago the Seminary chose Kartua, the open source ePortfolio tool.

Smith notes that every accredited institution has a series of standards from their accreditor that must be met.  NBTS is no different. The institution developed a series of learning outcomes to reflect those standards. “Every NBTS program has learning outcomes associated with it,” said Smith.  “And every course contains content that lines up with those learning outcomes.” By using Karuta, the institution can gather works or ‘artifacts’ from students demonstrating each
of those outcomes.

NBTS assigns two evaluators to each student who review the student’s artifacts in Karuta. “Our folks love being able to evaluate the artifacts (this way). You can examine the document online, and then it is just one-two-three, click, click, click.”  Evaluators can look at an artifact, evaluate it, and give it a score. “It’s wonderful,” Smith said.

Karuta is not being used so much for what Smith calls showcase portfolios.  “NBTS students are older, more stable,” she said. “Most of them already have jobs.”  They’re not as interested in using Karuta to demonstrate competency to prospective employers, even though the software has that capability.

In the years since NBTS went live with Karuta, “we have learned a lot, now that our folks are actually using Karuta. It was like pulling teeth (at first),” said Smith.  “But they’re using it now.” The institution has seen a more than five-fold increase in usage since the first year.

NBTS is now looking at adjusting its programs based on the data it is getting from Karuta. “The idea is to be able to tell which of our outcomes are being met, and which ones are we falling short on,” said Smith.  “We’re generating a lot of conversations with the faculty around this data.”

What’s next for New Brunswick Theological Seminary and Karuta?  Smith says that the institution is considering pursuing additional accreditations other than the ones it currently has. And with those new accreditations will come new standards, more learning outcomes, and additional artifacts to be gathered in Karuta.