公开(公告)号：US20170068659A1

公开(公告)日：2017-03-09

申请号：US15257217

申请日：2016-09-06

Applicant: VoiceBox Technologies Corporation

Inventor： Spencer John ROTHWELL ,  Daniela BRAGA ,  Ahmad Khamis ELSHENAWY ,  Stephen Steele CARTER

IPC: G06F17/27 ,  G06F17/24 ,  G10L15/22 ,  G10L15/26

CPC classification number: G10L15/22 ,  G06F17/273 ,  G06F17/278 ,  G06F17/30976 ,  G06Q10/10

Abstract: Systems and methods gathering text commands in response to a command context using a first crowdsourced are discussed herein. A command context for a natural language processing system may be identified, where the command context is associated with a command context condition to provide commands to the natural language processing system. One or more command creators associated with one or more command creation devices may be selected. A first application one the one or more command creation devices may be configured to display command creation instructions for each of the one or more command creators to provide text commands that satisfy the command context, and to display a field for capturing a user-generated text entry to satisfy the command creation condition in accordance with the command creation instructions. Systems and methods for reviewing the text commands using second and crowdsourced jobs are also presented herein.

Abstract translation: 这里讨论了使用第一个众包来响应于命令上下文收集文本命令的系统和方法。 可以识别自然语言处理系统的命令上下文，其中命令上下文与命令上下文条件相关联，以向自然语言处理系统提供命令。 可以选择与一个或多个命令创建设备相关联的一个或多个命令创建者。 可以将一个或多个命令创建设备的第一应用程序配置为显示用于一个或多个命令创建者中的每一个的命令创建指令以提供满足命令上下文的文本命令，并且显示用于捕获用户生成的文本的字段 根据命令创建指令来满足命令创建条件的条目。 本文还介绍了使用第二和众包作业查看文本命令的系统和方法。

公开(公告)号：US20170069039A1

公开(公告)日：2017-03-09

申请号：US15256930

申请日：2016-09-06

Applicant: VoiceBox Technologies Corporation

Inventor： Robert A. KENNEWICK ,  Spencer John ROTHWELL ,  John GILBERT

IPC: G06Q50/00 ,  G06Q30/02 ,  G06Q10/10

CPC classification number: G06Q50/01 ,  G06Q10/1097 ,  G06Q30/0218

Abstract: A system and method of characterizing crowd users that participate in crowd-sourced jobs based on responses to the jobs, and scheduling their participation based on user-indicated schedules of user availability or system-predicted schedules of user availability. A system may determine a level of quality of a response to a crowd job. The system may use the determined quality of response to determine a reward. The system may schedule a crowd user's participation in a future crowd job. The user may be identified based on the quality of previous responses provided by the user. The system may schedule the user's participation based on explicit input from the user indicating availability and/or based on a system-predicted availability of the user. When the future crowd job is or will be deployed, the system may provide the user with instructions to participate and/or otherwise provide the user with the crowd job.

Abstract translation: 基于对作业的响应来表征参与人群来源的工作的群众用户的系统和方法，并且基于用户指示的用户可用性计划或系统预测的用户可用性计划来安排他们的参与。 系统可以确定对人群工作的响应的质量水平。 系统可以使用确定的响应质量来确定奖励。 该系统可以安排人群用户参与未来的人群工作。 可以基于用户提供的先前响应的质量来识别用户。 系统可以基于来自用户的显式输入指示用户的参与和/或基于用户的系统预测的可用性来安排用户的参与。 当将来或将要部署未来的人群作业时，系统可以向用户提供参与和/或以其他方式向用户提供人群工作的指令。

公开(公告)号：US20170068809A1

公开(公告)日：2017-03-09

申请号：US15275720

申请日：2016-09-26

Applicant: VOICEBOX TECHNOLOGIES CORPORATION

Inventor： Vikrant BHOSALE ,  Spencer John ROTHWELL ,  Ahmad Khamis ELSHENAWY

IPC: G06F21/36

CPC classification number: G06F21/36 ,  G06F21/31 ,  G06F21/32 ,  G06F2221/2133

Abstract: The invention relates to a system and method of automatically distinguishing between computers and human based on responses to enhanced Completely Automated Public Turing test to tell Computers and Humans Apart (“e-captcha”) challenges that do not merely challenge the user to recognize skewed or stylized text. A given e-captcha challenge may be specific to a particular knowledge domain. Accordingly, e-captchas may be used not only to distinguish between computers and humans, but also determine whether a respondent has demonstrated knowledge in the particular knowledge domain. For instance, participants in crowd-sourced tasks, in which unmanaged crowds are asked to perform tasks, may be screened using an e-captcha challenge. This not only validates that a participant is a human (and not a bot, for example, attempting to game the crowd-source task), but also screens the participant based on whether they can successfully respond to the e-captcha challenge.

Abstract translation: 本发明涉及一种基于对增强的全自动公共图灵测试的响应来自动区分计算机和人的系统和方法，以告知计算机和人类（“e-captcha”）挑战，其不仅挑战用户识别偏斜或 风格化的文字。 给定的电子验证挑战可能是特定于知识领域的。 因此，e-captchas不仅可以用于区分计算机和人类，还可以确定答辩人是否已经在特定的知识领域中证明知识。 例如，可以使用电子验证挑战来筛选非托管人群执行任务的人群来源任务的参与者。 这不仅验证了参与者是人（而不是机器人，例如，尝试游戏人群来源任务），而且还可以根据是否能够成功应对电子验证挑战来筛选参与者。

公开(公告)号：US20170068651A1

公开(公告)日：2017-03-09

申请号：US15257084

申请日：2016-09-06

Applicant: VoiceBox Technologies Corporation

Inventor： Spencer John ROTHWELL ,  Daniela BRAGA ,  Ahmad Khamis ELSHENAWY ,  Stephen Steele CARTER

IPC: G06F17/24 ,  G06F17/27 ,  G06F17/21

CPC classification number: G06F17/241 ,  G06F17/218 ,  G06F17/278 ,  G06F17/2785

Abstract: A system and method of tagging utterances with Named Entity Recognition (“NER”) labels using unmanaged crowds is provided. The system may generate various annotation jobs in which a user, among a crowd, is asked to tag which parts of an utterance, if any, relate to various entities associated with a domain. For a given domain that is associated with a number of entities that exceeds a threshold N value, multiple batches of jobs (each batch having jobs that have a limited number of entities for tagging) may be used to tag a given utterance from that domain. This reduces the cognitive load imposed on a user, and prevents the user from having to tag more than N entities. As such, a domain with a large number of entities may be tagged efficiently by crowd participants without overloading each crowd participant with too many entities to tag.

Abstract translation: 提供了使用非托管人群使用命名实体识别（“NER”）标签来标记话语的系统和方法。 系统可以生成各种注释作业，其中在人群中的用户被要求标记话语的哪个部分（如果有的话）涉及与域相关联的各种实体。 对于与超过阈值N值的多个实体相关联的给定域，可以使用多批作业（每个批次具有具有有限数量的用于标记的实体的作业）来标记来自该域的给定话语。 这减少了施加在用户上的认知负荷，并且防止用户不必标记超过N个实体。 因此，具有大量实体的域可以被群众参与者有效地标记，而不会使具有太多实体的每个群众参与者超载以进行标记。

公开(公告)号：US20180277118A1

公开(公告)日：2018-09-27

申请号：US15924863

申请日：2018-03-19

Applicant: VOICEBOX TECHNOLOGIES CORPORATION

Inventor： Spencer John ROTHWELL ,  Daniela BRAGA ,  Ahmad Khamis ELSHENAWY ,  Stephen Steele CARTER

IPC: G10L15/30 ,  G10L15/01 ,  G06F17/28 ,  G06F17/27 ,  G06Q50/00 ,  G06Q30/02

CPC classification number: G10L15/30 ,  G06F17/2725 ,  G06F17/28 ,  G06Q30/02 ,  G06Q50/01 ,  G10L15/01

Abstract: Systems and methods of validating transcriptions of natural language content using crowdsourced validation jobs are provided herein. In various implementations, a transcription pair comprising natural language content and text corresponding to a transcription of the natural language content may be gathered. A group of validation devices may be selected for reviewing the transcription pair. A crowdsourced validation job may be created for the group of validation devices. The crowdsourced validation job may be provided to the group of validation devices. One or more votes representing whether or not the text accurately represents the natural language content may be received from the group of validation devices. Based on the one or more votes received, the transcription pair may be stored in a validated transcription library, which may be used to process end-user voice data.

公开(公告)号：US20170069326A1

公开(公告)日：2017-03-09

申请号：US15219088

申请日：2016-07-25

Applicant: VOICEBOX TECHNOLOGIES CORPORATION

Inventor： Spencer John ROTHWELL ,  Daniela BRAGA ,  Ahmad Khamis ELSHENAWY ,  Stephen Steele CARTER

IPC: G10L15/30 ,  G10L15/01 ,  G06F17/28 ,  G06F17/27

CPC classification number: G10L15/30 ,  G06F17/2725 ,  G06F17/28 ,  G06Q30/02 ,  G06Q50/01 ,  G10L15/01

Abstract: Systems and methods of validating transcriptions of natural language content using crowdsourced validation jobs are provided herein. In various implementations, a transcription pair comprising natural language content and text corresponding to a transcription of the natural language content may be gathered. A first group of validation devices may be selected for reviewing the transcription pair. A first crowdsourced validation job may be created for the first group of validation devices. The first crowdsourced validation job may be provided to the first group of validation devices. A vote representing whether or not the text accurately represents the natural language content may be received from each of the first group of validation devices. A validation score may be assigned to the transcription pair based, at least in part, on the votes from each of the first group of validation devices.

Abstract translation: 本文提供使用众包验证作业验证自然语言内容的转录的系统和方法。 在各种实现中，可以收集包括自然语言内容和对应于自然语言内容的转录的文本的转录对。 可以选择第一组验证装置来检查转录对。 可以为第一组验证设备创建第一个众包验证作业。 可以将第一群众化验证作业提供给第一组验证设备。 可以从第一组验证装置中的每一个接收表示文本是否准确地表示自然语言内容的投票。 至少部分地基于来自第一组验证装置的投票而将确认分数分配给转录对。

公开(公告)号：US20180121405A1

公开(公告)日：2018-05-03

申请号：US15675908

申请日：2017-08-14

Applicant: VoiceBox Technologies Corporation

Inventor： Spencer John ROTHWELL ,  Daniela BRAGA ,  Ahmad Khamis ELSHENAWY ,  Stephen Steele CARTER

IPC: G06F17/24 ,  G06F17/27 ,  G06F17/21

CPC classification number: G06F17/241 ,  G06F17/218 ,  G06F17/278 ,  G06F17/2785

Abstract: A system and method of tagging utterances with Named Entity Recognition (“NER”) labels using unmanaged crowds is provided. The system may generate various annotation jobs in which a user, among a crowd, is asked to tag which parts of an utterance, if any, relate to various entities associated with a domain. For a given domain that is associated with a number of entities that exceeds a threshold N value, multiple batches of jobs (each batch having jobs that have a limited number of entities for tagging) may be used to tag a given utterance from that domain. This reduces the cognitive load imposed on a user, and prevents the user from having to tag more than N entities. As such, a domain with a large number of entities may be tagged efficiently by crowd participants without overloading each crowd participant with too many entities to tag.

公开(公告)号：US20180033434A1

公开(公告)日：2018-02-01

申请号：US15728176

申请日：2017-10-09

Applicant: VoiceBox Technologies Corporation

Inventor： Spencer John ROTHWELL ,  Daniela BRAGA ,  Ahmad Khamis ELSHENAWY ,  Stephen Steele CARTER

IPC: G10L15/22 ,  G06F17/30 ,  G06F17/27 ,  G06Q10/10

CPC classification number: G10L15/22 ,  G06F16/90332 ,  G06F17/273 ,  G06F17/278 ,  G06Q10/10

Abstract: Systems and methods gathering text commands in response to a command context using a first crowdsourced are discussed herein. A command context for a natural language processing system may be identified, where the command context is associated with a command context condition to provide commands to the natural language processing system. One or more command creators associated with one or more command creation devices may be selected. A first application one the one or more command creation devices may be configured to display command creation instructions for each of the one or more command creators to provide text commands that satisfy the command context, and to display a field for capturing a user-generated text entry to satisfy the command creation condition in accordance with the command creation instructions. Systems and methods for reviewing the text commands using second and crowdsourced jobs are also presented herein.

公开(公告)号：US20170068656A1

公开(公告)日：2017-03-09

申请号：US15215114

申请日：2016-07-20

Applicant: VOICEBOX TECHNOLOGIES CORPORATION

Inventor： Daniela BRAGA ,  Spencer John ROTHWELL ,  Faraz ROMANI ,  Ahmad Khamis ELSHENAWY ,  Stephen Steele CARTER ,  Michael KENNEWICK

IPC: G06F17/27 ,  G10L17/24

CPC classification number: G06F17/278 ,  G06F17/2705 ,  G06F17/277 ,  G06Q10/10 ,  G10L15/26 ,  G10L17/24

Abstract: A system and method of recording utterances for building Named Entity Recognition (“NER”) models, which are used to build dialog systems in which a computer listens and responds to human voice dialog. Utterances to be uttered may be provided to users through their mobile devices, which may record the user uttering (e.g., verbalizing, speaking, etc.) the utterances and upload the recording to a computer for processing. The use of the user's mobile device, which is programmed with an utterance collection application (e.g., configured as a mobile app), facilitates the use of crowd-sourcing human intelligence tasking for widespread collection of utterances from a population of users. As such, obtaining large datasets for building NER models may be facilitated by the system and method disclosed herein.

Abstract translation: 记录用于构建命名实体识别（“NER”）模型的系统和方法，用于构建对话系统，其中计算机监听并响应人声对话。 可以通过其移动设备向用户提供要发出的话语，其可以记录用户发声（例如，言语，说话等）的话语，并将记录上传到计算机进行处理。 使用用话语收集应用（例如，配置为移动应用程序）编程的用户的移动设备有助于使用人群来源的人类智能任务来广泛地收集来自用户群体的话语。 因此，通过本文公开的系统和方法可以获得用于构建NER模型的大数据集。