python - jquery datatables with mongodb(mongoengine)

Question 1

Of course it is possible to use jQuery Datatables using your mentioned technologies. Pyramid is handling requests and manages database connection, processes requests into queries and assembles any query results or other data into a Response or return data that is given to renderers supporting templates written in Chameleon or Mako. You add jQuery DataTables to your templates - that's it from a bird's-eye view.

You have at least 2 ways to pass data to DataTables

assemble query results in a view, return data to a renderer using a chameleon template (http://docs.pylonsproject.org/projects/pyramid/en/1.5-branch/quick_tutorial/templating.html), use TAL to iterate over your data and generate valid DataTables HTML as shown in DataTables docs
if you finished version 1 switching to AJAX could be an option for you. Then you create DataTables markup in Javascript and use a Pyramid JSON renderer to return a JSON response to your AJAX client most probably using jQuery. (http://docs.pylonsproject.org/projects/pyramid/en/1.5-branch/quick_tutorial/json.html)

Your example is mixing up too much concerns and fails because of complexity. Take any official pyramid tutorial - you will hardly find any snippet that is that difficult to read and understand. Yours is difficult to read and understand - at least for me. Besides that it is not testable in any way. Pyramid starters should follow the development style shown. Pyramid really makes Unit & Functional Testing easy.

Follow this tutorial - it is worth the time: http://docs.pylonsproject.org/projects/pyramid/en/1.5-branch/quick_tutorial/index.html

Question 2

so I got it to work: I modified this script:

I created a module called tools and in the __init__.py:

from collections import namedtuple
import pymongo


# translation for sorting between datatables api and mongodb
order_dict = {'asc': 1, 'desc': -1}


class DataTables_Handler(object):
    def __init__(self, request, columns, index, collection):
        self.columns = columns
        self.index = index
        #self.db = get_db(request)
        self.collection = collection

        # values specified by the datatable for filtering, sorting, paging
        self.request_values = request.params

        self.db = pymongo.MongoClient()

        # results from the db
        self.result_data = None

        # total in the table after filtering
        self.cardinality_filtered = 0

        # total in the table unfiltered
        self.cadinality = 0

        self.run_queries()

    def output_result(self):
        output = {}
        output['sEcho'] = str(int(self.request_values['sEcho']))
        output['iTotalRecords'] = str(self.cardinality)
        output['iTotalDisplayRecords'] = str(self.cardinality_filtered)
        aaData_rows = []
        #print "sEcho", output['sEcho']

        for row in self.result_data:
            print row
            aaData_row = []
            for i in range(len(self.columns)):

                aaData_row.append(row[self.columns[i]])
                #print aData_row
            #add additional rows here that are not represented in the database
            aaData_row.append(('''<input id='%s' type='checkbox'></input>''' % (str(row[ self.index ]))).replace('\\', ''))

            aaData_rows.append(aaData_row)

        output['aaData'] = aaData_rows
        print "output: ", output
        return output

    def run_queries(self):
        #call db
        db = self.db.mydb

        # pages has 'start' and 'length' attributes
        pages = self.paging()

        # the term you entered into the datatable search
        filters = self.filtering()

        # the document field you chose to sort
        sorting = self.sorting()

        # get result from db
        self.result_data = db[self.collection].find(spec=filters,
                                                  skip=pages.start,
                                                  limit=pages.length,
                                                  sort=sorting)

        total_count = len(list(db[self.collection].find(spec=filters)))

        self.result_data = list(self.result_data)
        print "result_data", self.result_data

        self.cardinality_filtered = total_count

        self.cardinality = len(list(db[self.collection].find()))
        print "cardinality", self.cardinality

    def filtering(self):

        # build your filter spec
        filters = {}
        if ('sSearch' in self.request_values.keys()) and (self.request_values['sSearch'] != ""):

            # the term put into search is logically concatenated with 'or' between all columns
            or_filter_on_all_columns = []

            for i in range(len(self.columns)):
                column_filter = {}
                column_filter[self.columns[i]] = {'$regex':     self.request_values['sSearch'], '$options': 'i'}
            or_filter_on_all_columns.append(column_filter)
            filters['$or'] = or_filter_on_all_columns
        return filters

    def sorting(self):
        order = []
        # mongo translation for sorting order

        if (self.request_values['iSortCol_0'] != "") and (self.request_values['iSortingCols'] > 0):
            order = []
            for i in range(int(self.request_values['iSortingCols'])):
                order.append((self.columns[int(self.request_values['iSortCol_' + str(i)])], order_dict[self.request_values['sSortDir_' + str(i)]]))
        return order

    def paging(self):
        pages = namedtuple('pages', ['start', 'length'])
        if (self.request_values['iDisplayStart'] != "") and (self.request_values['iDisplayLength'] != -1):
            pages.start = int(self.request_values['iDisplayStart'])
            pages.length = int(self.request_values['iDisplayLength'])
        return pages

and in my View.py I have:

@view_config(
route_name='candidates.list.json',
renderer='json',
permission='admin'
)
def candidate_list_json(context, request):
    columns = [ '_id', 'first_name', 'email', 'mobile_number']
    index_column = "_id"
    collection = "candidates"

    results = DataTables_Handler(request, columns, index_column, collection).output_result()
    print "Results:   ", results
    # return the results as a string for the datatable
    return results

and then In my template I have:

<script src="{{'kivu:static/datatables/jquery.dataTables.js'|static_url}}"></script>
    <script src="{{'kivu:static/datatables/dataTables.scroller.js'|static_url}}"></script>   
<script>
    url = "{{'candidates.list.json'|route_url}}";
$(document).ready(function() {
    var oTable = $('#candidate_search').dataTable( {
        "sScrollY": "425px",
       // "sPaginationType": "full_numbers",
        "sAjaxSource": url,
        "bServerSide": true,
        "sDom": "<'row'<'col-xs-6 col-md-6 col-lg-6'><'col-xs-6 col-md-6 col-lg-6'f>r>t<'row'<'col-xs-6 col-md-6 col-lg-6'i><'col-xs-6 col-md-6 col-lg-6'>>S",
        //"sDom": "frtiS",
        //"sPaginationType": "bootstrap",
        "bProcessing" : true, 
       // "aoColumnDefs": [{ "bSortable": false, "aTargets": [ 5 ] }],
        "bDeferRender": true,
        "bStateSave": true
    } );
console.log(oTable);
        $.extend( $.fn.dataTableExt.oStdClasses, {
            "sSortAsc": "header headerSortDown",
            "sSortDesc": "header headerSortUp",
            "sSortable": "header"
        } );
        $.extend( $.fn.dataTableExt.oStdClasses, {
            "sWrapper": "dataTables_wrapper form-inline"
        } );
    } );</script>

I hope this can help someone, as class DataTables_Handler is re-useable.